Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #80571 > unrolled thread

How to write wide char string literals?

Back to article view | Back to comp.lang.c++

Contents

  How to write wide char string literals? Juha Nieminen <nospam@thanks.invalid> - 2021-06-30 07:57 +0000
    Re: How to write wide char string literals? Kli-Kla-Klawitter <kliklaklawitter69@gmail.com> - 2021-06-30 10:05 +0200
      Re: How to write wide char string literals? Ralf Goertz <me@myprovider.invalid> - 2021-06-30 10:30 +0200
        Re: How to write wide char string literals? Kli-Kla-Klawitter <kliklaklawitter69@gmail.com> - 2021-06-30 12:37 +0200
          Re: How to write wide char string literals? Ralf Goertz <me@myprovider.invalid> - 2021-07-01 09:19 +0200
            Re: How to write wide char string literals? Kli-Kla-Klawitter <kliklaklawitter69@gmail.com> - 2021-07-01 11:10 +0200
              Re: How to write wide char string literals? Ralf Goertz <me@myprovider.invalid> - 2021-07-01 11:36 +0200
                Re: How to write wide char string literals? Kli-Kla-Klawitter <kliklaklawitter69@gmail.com> - 2021-07-01 16:17 +0200
                  Re: How to write wide char string literals? Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-07-01 10:13 -0700
                    Re: How to write wide char string literals? Kli-Kla-Klawitter <kliklaklawitter69@gmail.com> - 2021-07-01 19:27 +0200
                      Re: How to write wide char string literals? Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-07-01 11:21 -0700
                        Re: How to write wide char string literals? Real Troll <real.troll@trolls.com> - 2021-07-01 18:45 +0000
                        Re: How to write wide char string literals? Kli-Kla-Klawitter <kliklaklawitter69@gmail.com> - 2021-07-02 10:03 +0200
                Re: How to write wide char string literals? Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-07-02 02:31 -0700
                  Re: How to write wide char string literals? MrSpud_oyCn@92wvlb1hltq4dhc.gov.uk - 2021-07-02 09:38 +0000
                    Re: How to write wide char string literals? Paavo Helde <myfirstname@osa.pri.ee> - 2021-07-02 13:52 +0300
                      Re: How to write wide char string literals? MrSpud_zg8yg@nx8574z6ey2rc0y563k5i2.net - 2021-07-02 12:53 +0000
                        Re: How to write wide char string literals? "Alf P. Steinbach" <alf.p.steinbach@gmail.com> - 2021-07-02 15:22 +0200
                        Re: How to write wide char string literals? Paavo Helde <myfirstname@osa.pri.ee> - 2021-07-02 17:32 +0300
    Re: How to write wide char string literals? Ralf Goertz <me@myprovider.invalid> - 2021-06-30 10:08 +0200
    Re: How to write wide char string literals? MrSpud_r5j@ywn9entw2s.org - 2021-06-30 08:39 +0000
      Re: How to write wide char string literals? Juha Nieminen <nospam@thanks.invalid> - 2021-06-30 10:52 +0000
        Re: How to write wide char string literals? MrSpud_3u59h8@0c9tv3nddl090w2ynhm.gov - 2021-06-30 11:28 +0000
          Re: How to write wide char string literals? David Brown <david.brown@hesbynett.no> - 2021-06-30 14:01 +0200
    Re: How to write wide char string literals? "Alf P. Steinbach" <alf.p.steinbach@gmail.com> - 2021-06-30 10:55 +0200
      Re: How to write wide char string literals? David Brown <david.brown@hesbynett.no> - 2021-06-30 11:23 +0200
    Re: How to write wide char string literals? Richard Damon <Richard@Damon-Family.org> - 2021-06-30 06:51 -0400
      Re: How to write wide char string literals? Juha Nieminen <nospam@thanks.invalid> - 2021-06-30 11:09 +0000
        Re: How to write wide char string literals? Richard Damon <Richard@Damon-Family.org> - 2021-06-30 07:35 -0400
    Re: How to write wide char string literals? Paavo Helde <myfirstname@osa.pri.ee> - 2021-06-30 14:03 +0300
      Re: How to write wide char string literals? Juha Nieminen <nospam@thanks.invalid> - 2021-06-30 11:13 +0000
        Re: How to write wide char string literals? Richard Damon <Richard@Damon-Family.org> - 2021-06-30 07:39 -0400
          Re: How to write wide char string literals? "Alf P. Steinbach" <alf.p.steinbach@gmail.com> - 2021-06-30 13:50 +0200
            Re: How to write wide char string literals? James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-06-30 14:19 -0400
              Re: How to write wide char string literals? "Alf P. Steinbach" <alf.p.steinbach@gmail.com> - 2021-07-01 13:31 +0200
          Re: How to write wide char string literals? Juha Nieminen <nospam@thanks.invalid> - 2021-07-01 04:42 +0000
            Re: How to write wide char string literals? James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-07-01 10:58 -0400
              Re: How to write wide char string literals? Juha Nieminen <nospam@thanks.invalid> - 2021-07-02 05:26 +0000
                Re: How to write wide char string literals? James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-07-02 01:44 -0400
                  Re: How to write wide char string literals? Bo Persson <bo@bo-persson.se> - 2021-07-02 08:33 +0200
                  Re: How to write wide char string literals? Juha Nieminen <nospam@thanks.invalid> - 2021-07-02 11:52 +0000
                    Re: How to write wide char string literals? James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-07-02 16:15 -0400
                      Re: How to write wide char string literals? "Alf P. Steinbach" <alf.p.steinbach@gmail.com> - 2021-07-03 03:30 +0200
                        Re: How to write wide char string literals? James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-07-03 00:44 -0400
                          Re: How to write wide char string literals? "Alf P. Steinbach" <alf.p.steinbach@gmail.com> - 2021-07-03 13:31 +0200
                            Re: How to write wide char string literals? James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-07-03 08:44 -0400
                              Re: How to write wide char string literals? "Alf P. Steinbach" <alf.p.steinbach@gmail.com> - 2021-07-03 16:28 +0200
                                Re: How to write wide char string literals? Richard Damon <Richard@Damon-Family.org> - 2021-07-03 11:48 -0400
                                Re: How to write wide char string literals? James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-07-08 15:56 -0400
                              Re: How to write wide char string literals? Juha Nieminen <nospam@thanks.invalid> - 2021-07-03 16:59 +0000
                                Re: How to write wide char string literals? James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-07-03 17:49 -0400
                                  Re: How to write wide char string literals? Juha Nieminen <nospam@thanks.invalid> - 2021-07-08 08:11 +0000
                                    Re: How to write wide char string literals? James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-07-08 05:52 -0400
                      Re: How to write wide char string literals? Juha Nieminen <nospam@thanks.invalid> - 2021-07-03 06:59 +0000
                        Re: How to write wide char string literals? James Kuyper <jameskuyper@alumni.caltech.edu> - 2021-07-03 09:03 -0400
                        Re: How to write wide char string literals? Chris Vine <chris@cvine--nospam--.freeserve.co.uk> - 2021-07-03 14:23 +0100
                          Re: How to write wide char string literals? Juha Nieminen <nospam@thanks.invalid> - 2021-07-03 17:06 +0000
                            Re: How to write wide char string literals? Richard Damon <Richard@Damon-Family.org> - 2021-07-03 14:06 -0400
                              Re: How to write wide char string literals? Juha Nieminen <nospam@thanks.invalid> - 2021-07-08 08:13 +0000
                  Re: How to write wide char string literals? "Alf P. Steinbach" <alf.p.steinbach@gmail.com> - 2021-07-02 15:37 +0200
        Re: How to write wide char string literals? Manfred <noname@add.invalid> - 2021-06-30 16:54 +0200
        Re: How to write wide char string literals? Paavo Helde <myfirstname@osa.pri.ee> - 2021-06-30 17:56 +0300
    Re: How to write wide char string literals? Öö Tiib <ootiib@hot.ee> - 2021-06-30 09:22 -0700
    Re: How to write wide char string literals? Christian Gollwitzer <auriocus@gmx.de> - 2021-07-01 07:19 +0200
      Re: How to write wide char string literals? David Brown <david.brown@hesbynett.no> - 2021-07-01 10:29 +0200
        Re: How to write wide char string literals? Juha Nieminen <nospam@thanks.invalid> - 2021-07-01 08:44 +0000
          Re: How to write wide char string literals? David Brown <david.brown@hesbynett.no> - 2021-07-01 12:58 +0200
        Re: How to write wide char string literals? Christian Gollwitzer <auriocus@gmx.de> - 2021-07-01 14:01 +0200
          Re: How to write wide char string literals? David Brown <david.brown@hesbynett.no> - 2021-07-01 14:57 +0200
      Re: How to write wide char string literals? Manfred <noname@add.invalid> - 2021-07-01 15:44 +0200

Page 4 of 4 — ← Prev page 1 2 3 [4]

#80589

On 6/30/2021 1:13 PM, Juha Nieminen wrote:
> Paavo Helde <myfirstname@osa.pri.ee> wrote:
>> I want my code to work anywhere with any compiler/framework conventions
>> and settings, and all my internal strings are in UTF-8 anyway, so I can
>> use strict ASCII source files with hardcoded UTF-8 characters, e.g.:
>>
>> std::string s = "Copyright \xC2\xA9 2001-2020";
> 
> Does that work for wide string literals? Because I don't think it does.
> In other words:
> 
> std::wstring s = L"Copyright \xC2\xA9 2001-2020";
> 
> However, as suggested in another reply, using "\uXXXX" instead ought
> to work just fine (regardless of whether it's a narrow or wide char
> literal).

Yes, it is mandated by the standard. Ref. "universal-character-name"

  As long as you don't need the readability, of course.
> 

But you have no readability with "\xHH" either, do you?

[toc] | [prev] | [next] | [standalone]

#80590

30.06.2021 14:13 Juha Nieminen kirjutas:
> Paavo Helde <myfirstname@osa.pri.ee> wrote:
>> I want my code to work anywhere with any compiler/framework conventions
>> and settings, and all my internal strings are in UTF-8 anyway, so I can
>> use strict ASCII source files with hardcoded UTF-8 characters, e.g.:
>>
>> std::string s = "Copyright \xC2\xA9 2001-2020";
> 
> Does that work for wide string literals? Because I don't think it does.
> In other words:
> 
> std::wstring s = L"Copyright \xC2\xA9 2001-2020";
> 
> However, as suggested in another reply, using "\uXXXX" instead ought
> to work just fine (regardless of whether it's a narrow or wide char
> literal). As long as you don't need the readability, of course.

Sure, you can use \x also in wide strings. However, as a wide string is 
not an 8-bit encoding, one should not use UTF-8 encoding, but either 
UTF-16 or UTF-32, depending on sizeof(wchar_t).

A working example for Windows/MSVC++ with UTF-16 and sizeof(wchar_t)==2:


#include <Windows.h>
#include <string>

int main() {

   std::wstring s =
     L"Copyright \xA9 2001-2020\r\n"
     L"The smallest infinite cardinal number is \x2080\x5d0\r\n"
     L"A single hieroglyph not fitting in 16 bits: \xD840\xDC0F";

   ::MessageBoxW(nullptr, s.c_str(), L"Test", MB_OK);
}

This would not be portable to platforms where sizeof(wchar_t)==4. As far 
as I gather, the \u and \U escapes ought to be more portable.

[toc] | [prev] | [next] | [standalone]

#80591

On Wednesday, 30 June 2021 at 10:57:30 UTC+3, Juha Nieminen wrote:
> Character encoding was a problem in the 1960's, and it's still a problem 
> today, no matter how much computers advance. Sheesh. 
> 
> Problem is, how to reliably write wide char string literals that contain 
> non-ascii characters? 
> 
> Suppose you write for example this: 
> 
> const wchar_t* str = L"???"; 
> 
> In the *source code* that string literal may be eg. UTF-8 encoded. However, 
> the compiler needs to convert it to wide chars. 
> 
> Problem is, how does the compiler know which encoding is being used in 
> that 8-bit string literal in the source code, in order for it to convert 
> it properly to wide chars? 

By telling it to compiler. Or to IDE that deals with compiler. 

> Some compilers may assume it's UTF-8 encoded source code. Others may 
> assume it's ISO-Latin-1 encoded (I'm looking at you, Visual Studio). 
> Obviously the end result will be garbage if the wrong assumption is made. 

 It was already in VS2008 something like ...open the file in VS, File->Advanced
Save Options then “Encoding” combo let to select UTF-8.

> In most compilers (such as Visual Studio) you can specify which encoding 
> to assume for source files, but this has to be done at the project 
> settings level. I don't think there's any way to specify the encoding 
> in the source code itself.

Maybe some compilers examine BOM but I have no knowledge there.
All of source files I keep in UTF-8. That does not need BOM. If I get some
UTF-16 or UTF-32 file then I turn it into UTF-8 anyway first before committing 
anywhere.
 
> What does the C++ standard say? Does it say that source code files are 
> always UTF-8 encoded, or is it up to the implementation? I assume that if 
> it's the latter, the standard doesn't provide any mechanism to specify 
> which encoding is being used. Or does it?

Compiler's command line is not standardized yet.

[toc] | [prev] | [next] | [standalone]

#80594

Am 30.06.21 um 09:57 schrieb Juha Nieminen:
> Character encoding was a problem in the 1960's, and it's still a problem
> today, no matter how much computers advance. Sheesh.
> 
> Problem is, how to reliably write wide char string literals that contain
> non-ascii characters?
> 
> Suppose you write for example this:
> 
>      const wchar_t* str = L"???";
> 
> In the *source code* that string literal may be eg. UTF-8 encoded. However,
> the compiler needs to convert it to wide chars.

I think it is best to avoid wide strings. Now that doesn't help you if 
you need them to call native Windows functions which insist on wchar_t.

I'm still wondering why you need to put a Unicode string in the source 
code at all. Could you use an i18n feature of Windows to lookup the real 
string? I'm not an expert of i18n on Windows, but using GNU gettext, you 
would write some ASCII equivalent thing in the code and then have an 
auxiliary translation file with a well defined encoding. At runtime the 
ASCII string is merely a key into the table. Plus the added bonus that 
you can support multiple languages.

	Christian

[toc] | [prev] | [next] | [standalone]

#80596

On 01/07/2021 07:19, Christian Gollwitzer wrote:
> Am 30.06.21 um 09:57 schrieb Juha Nieminen:
>> Character encoding was a problem in the 1960's, and it's still a problem
>> today, no matter how much computers advance. Sheesh.
>>
>> Problem is, how to reliably write wide char string literals that contain
>> non-ascii characters?
>>
>> Suppose you write for example this:
>>
>>      const wchar_t* str = L"???";
>>
>> In the *source code* that string literal may be eg. UTF-8 encoded.
>> However,
>> the compiler needs to convert it to wide chars.
> 
> I think it is best to avoid wide strings. Now that doesn't help you if
> you need them to call native Windows functions which insist on wchar_t.
> 
> I'm still wondering why you need to put a Unicode string in the source
> code at all. Could you use an i18n feature of Windows to lookup the real
> string? I'm not an expert of i18n on Windows, but using GNU gettext, you
> would write some ASCII equivalent thing in the code and then have an
> auxiliary translation file with a well defined encoding. At runtime the
> ASCII string is merely a key into the table. Plus the added bonus that
> you can support multiple languages.
> 
>     Christian

Code can require non-ASCII characters without need internationalisation.
 gettext and the like are certainly useful, but they are very heavy
tools compared to a fixed string or small table of strings in the code.
 If you are writing a program for use in single company in Germany
(since you have a German email address), with all the texts in German,
would you want to use internationalisation frameworks just to make
"groß" turn out right?

The OP could also be working on embedded systems or some other code for
which having a single self-contained executable is important.

[toc] | [prev] | [next] | [standalone]

#80597

David Brown <david.brown@hesbynett.no> wrote:
> Code can require non-ASCII characters without need internationalisation.
>  gettext and the like are certainly useful, but they are very heavy
> tools compared to a fixed string or small table of strings in the code.
>  If you are writing a program for use in single company in Germany
> (since you have a German email address), with all the texts in German,
> would you want to use internationalisation frameworks just to make
> "groß" turn out right?

There are also many situations where using non-ascii characters in
string literals may not be related to language and internationalization.
After all, Unicode contains loads of characters that are not related
to spoken languages, such as math symbols, and lots of other types
of symbols which are universal and don't require any sort of
internationalization. Sometimes these symbols may be used all on
their own, sometimes as part of text (eg. in labels and titles).

Also, unit tests for code supporting Unicode may benefit from
being able to use string literals with non-ascii characters.
(Of course, as noted in other posts in this thread, there is a
working solution to get around this, and it's the use of the \u
escape character.)

[toc] | [prev] | [next] | [standalone]

#80600

On 01/07/2021 10:44, Juha Nieminen wrote:
> David Brown <david.brown@hesbynett.no> wrote:
>> Code can require non-ASCII characters without need internationalisation.
>>  gettext and the like are certainly useful, but they are very heavy
>> tools compared to a fixed string or small table of strings in the code.
>>  If you are writing a program for use in single company in Germany
>> (since you have a German email address), with all the texts in German,
>> would you want to use internationalisation frameworks just to make
>> "groß" turn out right?
> 
> There are also many situations where using non-ascii characters in
> string literals may not be related to language and internationalization.

Good point.

> After all, Unicode contains loads of characters that are not related
> to spoken languages, such as math symbols, and lots of other types
> of symbols which are universal and don't require any sort of
> internationalization. Sometimes these symbols may be used all on
> their own, sometimes as part of text (eg. in labels and titles).
> 
> Also, unit tests for code supporting Unicode may benefit from
> being able to use string literals with non-ascii characters.
> (Of course, as noted in other posts in this thread, there is a
> working solution to get around this, and it's the use of the \u
> escape character.)
> 

Yes - but such workarounds are hideous compared to writing:

	printf("Temperature %.1f °C\n", 123.4);


I am glad most of my code only has to compile with gcc, and I can ignore
such portability matters.

[toc] | [prev] | [next] | [standalone]

#80602

Am 01.07.21 um 10:29 schrieb David Brown:
> 
> Code can require non-ASCII characters without need internationalisation.
>   gettext and the like are certainly useful, but they are very heavy
> tools compared to a fixed string or small table of strings in the code >   If you are writing a program for use in single company in Germany
> (since you have a German email address), with all the texts in German,
> would you want to use internationalisation frameworks just to make
> "groß" turn out right?

I can see your point, but actually, most programs developed here in 
Germany are still written in English. This is true for comments and 
variable names etc., because there might be a non-German coworker 
involved, and mostly because people are simply used to English as "the 
computer language". I've seen German comments, variable names and 
literal strings only at the university in introductory programming 
courses etc. But admittedly, I never produced GUIs in C++, because there 
are easier options available - and these usually come with good Unicode 
support (e.g. Python). These smaller tools did not get i18n'ed.
I still think that if I had to make a program with a German interface, 
it would make sense to write it with English strings and translate it 
with a tool - because then, adding French or Turkish later on would be 
easy.

> 
> The OP could also be working on embedded systems or some other code for
> which having a single self-contained executable is important.
> 

OK yes there are certainly points where this approach is not suitable. 
Just wanted to bring another solution to the table.

Ceterum censeo wchar_t esse inutilem ;)

	Christian

[toc] | [prev] | [next] | [standalone]

#80603

On 01/07/2021 14:01, Christian Gollwitzer wrote:
> Am 01.07.21 um 10:29 schrieb David Brown:
>>
>> Code can require non-ASCII characters without need internationalisation.
>>   gettext and the like are certainly useful, but they are very heavy
>> tools compared to a fixed string or small table of strings in the code
>> >   If you are writing a program for use in single company in Germany
>> (since you have a German email address), with all the texts in German,
>> would you want to use internationalisation frameworks just to make
>> "groß" turn out right?
> 
> I can see your point, but actually, most programs developed here in
> Germany are still written in English. This is true for comments and
> variable names etc., because there might be a non-German coworker
> involved, and mostly because people are simply used to English as "the
> computer language". I've seen German comments, variable names and
> literal strings only at the university in introductory programming
> courses etc. 

Sure.  The same is true here in Norway.  But it is not true everywhere.
 And even when you have English identifiers, comments, etc., the text
strings you show to users are often in a language other than English.

> But admittedly, I never produced GUIs in C++, because there
> are easier options available - and these usually come with good Unicode
> support (e.g. Python). These smaller tools did not get i18n'ed.

I do that too.

> I still think that if I had to make a program with a German interface,
> it would make sense to write it with English strings and translate it
> with a tool - because then, adding French or Turkish later on would be
> easy.

Most software is written for one or a few customers, and one language
will always be sufficient.  Of course, such software is also almost
always written for one compiler and one target, and portability of
source code is not an issue.  This is particularly true if you have good
modularisation in the code - the user-facing parts with the text strings
are less likely to be re-used elsewhere than the more library-like code
underneath.

> 
>>
>> The OP could also be working on embedded systems or some other code for
>> which having a single self-contained executable is important.
>>
> 
> OK yes there are certainly points where this approach is not suitable.
> Just wanted to bring another solution to the table.
> 
> Ceterum censeo wchar_t esse inutilem ;)
> 

Agreed - and salt the ground it was built on, to save future generations
from its curse!

[toc] | [prev] | [next] | [standalone]

#80604

On 7/1/2021 7:19 AM, Christian Gollwitzer wrote:
> Am 30.06.21 um 09:57 schrieb Juha Nieminen:
>> Character encoding was a problem in the 1960's, and it's still a problem
>> today, no matter how much computers advance. Sheesh.
>>
>> Problem is, how to reliably write wide char string literals that contain
>> non-ascii characters?
>>
>> Suppose you write for example this:
>>
>>      const wchar_t* str = L"???";
>>
>> In the *source code* that string literal may be eg. UTF-8 encoded. 
>> However,
>> the compiler needs to convert it to wide chars.
> 
> I think it is best to avoid wide strings. Now that doesn't help you if 
> you need them to call native Windows functions which insist on wchar_t.
> 
> I'm still wondering why you need to put a Unicode string in the source 
> code at all. Could you use an i18n feature of Windows to lookup the real 
> string? I'm not an expert of i18n on Windows, but using GNU gettext, you 
> would write some ASCII equivalent thing in the code and then have an 
> auxiliary translation file with a well defined encoding. At runtime the 
> ASCII string is merely a key into the table. Plus the added bonus that 
> you can support multiple languages.

In Windows programming the need for wchar_t strings is relatively 
common, since this its native character set. Most APIs are provided with 
both ASCII and WCHAR variants, however if you need more-than-plain-ascii 
text support you almost invariably end up #define'ing UNICODE and thus 
default to the wide char variants - moreover some APIs are given for 
WCHAR strings only.
In these cases if you need strings literals they are best expressed as 
WCHAR strings directly to avoid unnecessary conversions at runtime.


> 
>      Christian

[toc] | [prev] | [standalone]

Page 4 of 4 — ← Prev page 1 2 3 [4]

Back to top | Article view | comp.lang.c++

csiph-web