Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #82564

Character encoding conversion in wide string literals

Show all headers | View raw

Recently I stumbled across a problem where I had wide string literals
with non-ascii characters UTF-8 encoded. In other words, I had code like
this (I'm using non-ascii in the code below, I hope it doesn't get
mangled up, but even if it does, it should nevertheless be clear what
I'm trying to express):

    std::wstring str = L"non-ascii chars: ???";

The C++ source file itself uses UTF-8 encoding, meaning that that line
of code is likewise UTF-8 encoded. If it were a narrow string literal
(being assigned to a std::string) then it works just fine (primarily
because the compiler doesn't need to do anything to it, it can simply
take those bytes from the source file as is).

However, since it's a wide string literal (being assigned to a std::wstring)
it's not as clear-cut anymore. What does the standard say about this
situation?

The thing is that it works just fine in Linux using gcc. The compiler will
re-encode the UTF-8 encoded characters in the source file inside the
parentheses into whatever encoding wide char string use, so the correct
content will end up in the executable binary (and thus in the wstring).

Apparently it does not work correctly in (some recent version of)
Visual Studio, where apparently it just takes the byte values from the
source file within the parentheses as-is, and just assigns those values
as-is to the wide chars that end up in the binary. (Or something like that.)

Does the standard specify what the compiler should do in this situation?
If not, then what is the proper way of specifying wide string literals
that contain non-ascii characters?

Back to comp.lang.c++ | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

Character encoding conversion in wide string literals Juha Nieminen <nospam@thanks.invalid> - 2021-12-07 11:37 +0000
  Re: Character encoding conversion in wide string literals "Alf P. Steinbach" <alf.p.steinbach@gmail.com> - 2021-12-07 16:44 +0100
    Re: Character encoding conversion in wide string literals James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-12-07 11:48 -0500
      Re: Character encoding conversion in wide string literals "Alf P. Steinbach" <alf.p.steinbach@gmail.com> - 2021-12-07 18:59 +0100
        Re: Character encoding conversion in wide string literals James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-12-07 13:55 -0500
        Re: Character encoding conversion in wide string literals Öö Tiib <ootiib@hot.ee> - 2021-12-07 16:39 -0800
        Re: Character encoding conversion in wide string literals David Brown <david.brown@hesbynett.no> - 2021-12-08 09:18 +0100
    Re: Character encoding conversion in wide string literals Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-07 09:41 -0800
      Re: Character encoding conversion in wide string literals "Alf P. Steinbach" <alf.p.steinbach@gmail.com> - 2021-12-07 19:07 +0100
        Re: Character encoding conversion in wide string literals Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-07 12:21 -0800
      Re: Character encoding conversion in wide string literals Paavo Helde <eesnimi@osa.pri.ee> - 2021-12-07 20:32 +0200
        Re: Character encoding conversion in wide string literals James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-12-07 13:55 -0500
        Re: Character encoding conversion in wide string literals Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-07 12:26 -0800
  Re: Character encoding conversion in wide string literals James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-12-07 11:38 -0500
    Re: Character encoding conversion in wide string literals Manfred <noname@add.invalid> - 2021-12-07 19:07 +0100

csiph-web