Path: csiph.com!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail From: Keith Thompson Newsgroups: comp.lang.c++ Subject: Re: Character encoding conversion in wide string literals Date: Tue, 07 Dec 2021 12:21:47 -0800 Organization: None to speak of Lines: 80 Message-ID: <87mtlcb1yc.fsf@nosuchdomain.example.com> References: <87r1aob9d3.fsf@nosuchdomain.example.com> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: reader02.eternal-september.org; posting-host="7f90512ec77176d2b122f800cb8bb3bb"; logging-data="17099"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1/7jrEpNBNKCrn3HYyirg1y" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) Cancel-Lock: sha1:nY/gCUdK/DI2PDL/XKQN9vJjGBs= sha1:gvsueIEfag3nGxMci+ZxRUZZz0w= Xref: csiph.com comp.lang.c++:82581 "Alf P. Steinbach" writes: > On 7 Dec 2021 18:41, Keith Thompson wrote: >> "Alf P. Steinbach" writes: >>> On 7 Dec 2021 12:37, Juha Nieminen wrote: >>>> Recently I stumbled across a problem where I had wide string literals >>>> with non-ascii characters UTF-8 encoded. In other words, I had code like >>>> this (I'm using non-ascii in the code below, I hope it doesn't get >>>> mangled up, but even if it does, it should nevertheless be clear what >>>> I'm trying to express): >>>> std::wstring str = L"non-ascii chars: ???"; >>>> The C++ source file itself uses UTF-8 encoding, meaning that that >>>> line >>>> of code is likewise UTF-8 encoded. If it were a narrow string literal >>>> (being assigned to a std::string) then it works just fine (primarily >>>> because the compiler doesn't need to do anything to it, it can simply >>>> take those bytes from the source file as is). >>>> However, since it's a wide string literal (being assigned to a >>>> std::wstring) >>>> it's not as clear-cut anymore. What does the standard say about this >>>> situation? >>>> The thing is that it works just fine in Linux using gcc. The >>>> compiler will >>>> re-encode the UTF-8 encoded characters in the source file inside the >>>> parentheses into whatever encoding wide char string use, so the correct >>>> content will end up in the executable binary (and thus in the wstring). >>>> Apparently it does not work correctly in (some recent version of) >>>> Visual Studio, where apparently it just takes the byte values from the >>>> source file within the parentheses as-is, and just assigns those values >>>> as-is to the wide chars that end up in the binary. (Or something like that.) >>> >>> The Visual C++ compiler assumes that source code is Windows ANSI >>> encoded unless >>> >>> * you use an encoding option such as `/utf8`, or >>> * the source is UTF-8 with BOM, or >>> * the source is UTF-16. >> What exactly do you mean by "Windows ANSI"? Windows-1252 or >> something >> else? (Microsoft doesn't call it "ANSI", because it isn't.) >> [...] > > "Windows ANSI" is the encoding specified by the `GetACP` API function, > which, but as I recall that's more or less undocumented, just serves > up the codepage number specified by registry value > > Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage@ACP > > This means that "Windows ANSI" is a pretty dynamic thing. Not just > system-dependent, but at-the-moment-configuration dependent. Though in > English-speaking countries it's Windows 1252 by default. > > And that in turn means that using the defaults with Visual C++, you > can end up with pretty much any encoding whatsoever of narrow > literals. > > Which means that it's a good idea to take charge. > > Option `/utf8` is one way to take charge. It appears my previous statement was incorrect. At least some Microsoft documentation does still (incorrectly) refer to "Windows ANSI". https://docs.microsoft.com/en-us/windows/win32/api/winnls/nf-winnls-getacp The history, as I recall, is that Microsoft proposed one or more 8-bit extensions of the 7-bit ASCII character set as ANSI standards. Windows-1252, which has various accented letters and other symbols in the range 128-255, is the best known variant. But Microsoft's proposal was never adopted by ANSI, leaving us with a bunch of incorrect documentation. Instead, ISO created the 8859-* 8-bit character sets, including 8859-1, or Latin-1. Latin-1 differs from Windows-1252 in that Latin-1 it has control characters in the range 128-159, while Windows-1252 has printable characters. https://en.wikipedia.org/wiki/Windows-1252 -- Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com Working, but not speaking, for Philips void Void(void) { Void(); } /* The recursive call of the void */