Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #44552 > unrolled thread

"Unicode"

Back to article view | Back to comp.lang.c++

Contents

  "Unicode" Lynn McGuire <lynnmcguire5@gmail.com> - 2016-08-29 12:06 -0500
    Re: "Unicode" Daniel <danielaparker@gmail.com> - 2016-08-29 11:52 -0700
      Re: "Unicode" Lynn McGuire <lynnmcguire5@gmail.com> - 2016-08-30 12:24 -0500
        Re: "Unicode" legalize+jeeves@mail.xmission.com (Richard) - 2016-08-30 18:50 +0000
          Re: "Unicode" Lynn McGuire <lynnmcguire5@gmail.com> - 2016-08-30 14:11 -0500
            Re: "Unicode" legalize+jeeves@mail.xmission.com (Richard) - 2016-08-30 19:44 +0000
          Re: "Unicode" Vir Campestris <vir.campestris@invalid.invalid> - 2016-08-30 21:29 +0100
            Re: "Unicode" legalize+jeeves@mail.xmission.com (Richard) - 2016-08-31 20:31 +0000
              Re: "Unicode" "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> - 2016-09-01 01:50 +0200
                Re: "Unicode" Daniel <danielaparker@gmail.com> - 2016-08-31 18:47 -0700
                  Re: "Unicode" "Osmium" <r124c4u102@comcast.net> - 2016-09-01 06:01 -0500
                Re: "Unicode" Jerry Stuckle <jstucklex@attglobal.net> - 2016-08-31 23:06 -0400
                  Re: "Unicode" Ian Collins <ian-news@hotmail.com> - 2016-09-01 15:21 +1200
                    Re: "Unicode" Jerry Stuckle <jstucklex@attglobal.net> - 2016-09-01 15:07 -0400
                  Re: "Unicode" "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> - 2016-09-01 08:02 +0200
                    Re: "Unicode" legalize+jeeves@mail.xmission.com (Richard) - 2016-09-01 18:46 +0000
                  Re: "Unicode" David Brown <david.brown@hesbynett.no> - 2016-09-01 10:09 +0200
                  Re: "Unicode" Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> - 2016-09-01 17:58 +0100
                    Re: "Unicode" Jerry Stuckle <jstucklex@attglobal.net> - 2016-09-01 15:08 -0400
                      Re: "Unicode" Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> - 2016-09-01 21:20 +0100
                        Re: "Unicode" Jerry Stuckle <jstucklex@attglobal.net> - 2016-09-01 20:52 -0400
                          Re: "Unicode" Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> - 2016-09-02 14:44 +0100
                            Re: "Unicode" Jerry Stuckle <jstucklex@attglobal.net> - 2016-09-02 10:10 -0400
                              Re: "Unicode" Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> - 2016-09-02 16:35 +0100
                                Re: "Unicode" Jerry Stuckle <jstucklex@attglobal.net> - 2016-09-02 12:26 -0400
                              Re: "Unicode" Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> - 2016-09-02 17:02 +0100
                                Re: "Unicode" Jerry Stuckle <jstucklex@attglobal.net> - 2016-09-02 12:33 -0400
                                  Re: "Unicode" Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> - 2016-09-02 17:53 +0100
                                    Re: "Unicode" Gareth Owen <gwowen@gmail.com> - 2016-09-02 19:20 +0100
                                    Re: "Unicode" Jerry Stuckle <jstucklex@attglobal.net> - 2016-09-02 16:36 -0400
                                      Re: "Unicode" Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> - 2016-09-02 21:45 +0100
                                        Re: "Unicode" Jerry Stuckle <jstucklex@attglobal.net> - 2016-09-02 19:57 -0400
                Re: "Unicode" legalize+jeeves@mail.xmission.com (Richard) - 2016-09-01 18:45 +0000
                  Re: "Unicode" Jerry Stuckle <jstucklex@attglobal.net> - 2016-09-01 15:11 -0400
                  Re: "Unicode" David Brown <david.brown@hesbynett.no> - 2016-09-01 22:27 +0200
                    Re: "Unicode" Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> - 2016-09-01 21:37 +0100
                    Re: "Unicode" Jerry Stuckle <jstucklex@attglobal.net> - 2016-09-01 20:53 -0400
                  Re: "Unicode" "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> - 2016-09-02 05:57 +0200
        Re: "Unicode" Bo Persson <bop@gmb.dk> - 2016-08-30 21:30 +0200
          Re: "Unicode" Bo Persson <bop@gmb.dk> - 2016-08-30 21:40 +0200
          Re: "Unicode" Daniel <danielaparker@gmail.com> - 2016-08-30 13:39 -0700
    Re: "Unicode" Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> - 2016-09-01 21:54 +0100
      Re: "Unicode" raltbos@xs4all.nl (Richard Bos) - 2016-09-03 08:00 +0000
        Re: "Unicode" Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> - 2016-09-03 14:10 -0500
          Re: "Unicode" "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> - 2016-09-04 09:56 +0200
            Re: "Unicode" Paavo Helde <myfirstname@osa.pri.ee> - 2016-09-04 14:37 +0300
              Re: "Unicode" "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> - 2016-09-05 12:53 +0200
                Re: "Unicode" Paavo Helde <myfirstname@osa.pri.ee> - 2016-09-05 15:30 +0300
                  Re: "Unicode" "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> - 2016-09-05 16:13 +0200
                    Re: "Unicode" Paavo Helde <myfirstname@osa.pri.ee> - 2016-09-05 22:03 +0300
                      Re: "Unicode" "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> - 2016-09-05 23:54 +0200
                        Re: "Unicode" Keith Thompson <kst-u@mib.org> - 2016-09-05 15:35 -0700
                          Re: "Unicode" "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> - 2016-09-06 00:18 +0200
                            Re: "Unicode" Keith Thompson <kst-u@mib.org> - 2016-09-05 16:17 -0700
                              Re: "Unicode" "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> - 2016-09-06 01:20 +0200
                                Re: "Unicode" Keith Thompson <kst-u@mib.org> - 2016-09-05 17:16 -0700
                                  Re: "Unicode" "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> - 2016-09-06 02:42 +0200
                        Re: "Unicode" Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> - 2016-09-05 23:05 +0100
            Re: "Unicode" Vir Campestris <vir.campestris@invalid.invalid> - 2016-09-04 21:37 +0100
              Re: "Unicode" Paavo Helde <myfirstname@osa.pri.ee> - 2016-09-05 00:22 +0300
                Re: "Unicode" Vir Campestris <vir.campestris@invalid.invalid> - 2016-09-05 21:32 +0100
                  Re: "Unicode" Paavo Helde <myfirstname@osa.pri.ee> - 2016-09-06 00:00 +0300
                    Re: "Unicode" Vir Campestris <vir.campestris@invalid.invalid> - 2016-09-07 21:19 +0100
                      Re: "Unicode" "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> - 2016-09-08 02:16 +0200

Page 3 of 4 — ← Prev page 1 2 [3] 4  Next page →

#44621

On Tuesday, August 30, 2016 at 3:30:47 PM UTC-4, Bo Persson wrote:
> On 2016-08-30 19:24, Lynn McGuire wrote:
> >
> > Microsoft has a big hand in the screw up with their decision to go with
> > wide characters.
> 
> The decision was pretty obvious at the time, when their wchar_t could 
> hold all of the existing code space. Who would know that even-wider 
> characters would appear later?
> 
Computer languages should have string types that have string semantics. C++ doesn't, std::string is just an array of bytes, std::wstring is just an array of, well, pick your operating system. Neither of these instantiations of basic_string offers much of anything over any other sequence type in the language. Users have to be intimately aware of the character encoding, they shouldn't have to be.

Daniel

[toc] | [prev] | [next] | [standalone]

#44708

On 29/08/2016 18:06, Lynn McGuire wrote:
> "Unicode"
>    http://xkcd.com/1726/
>
> "Watching the Unicode people try to govern the infinite chaos of human
> language with consistent technical standards is like watching highway
> engineers try to steer a river using traffic signs."
>
> As usual, Randall Munroe nails it.

Unicode is simple in C++: just use std::string and UTF-8.

/Flibble

[toc] | [prev] | [next] | [standalone]

#44778

Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> wrote:

> On 29/08/2016 18:06, Lynn McGuire wrote:
> > "Unicode"
> >    http://xkcd.com/1726/
> >
> > "Watching the Unicode people try to govern the infinite chaos of human
> > language with consistent technical standards is like watching highway
> > engineers try to steer a river using traffic signs."
> >
> > As usual, Randall Munroe nails it.
> 
> Unicode is simple in C++: just use std::string and UTF-8.

You... do not understand Unicode.

If you think the problem is just one of choosing one correct charset,
you're very naive. The problems are much hairier than that, and much
more detailed.

Richard

[toc] | [prev] | [next] | [standalone]

#44787

Richard Bos <raltbos@xs4all.nl> wrote:
> Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> wrote:
> 
>> On 29/08/2016 18:06, Lynn McGuire wrote:
>>> "Unicode"
>>> http://xkcd.com/1726/
>>> 
>>> "Watching the Unicode people try to govern the infinite chaos of human
>>> language with consistent technical standards is like watching highway
>>> engineers try to steer a river using traffic signs."
>>> 
>>> As usual, Randall Munroe nails it.
>> 
>> Unicode is simple in C++: just use std::string and UTF-8.
> 
> You... do not understand Unicode.
> 
> If you think the problem is just one of choosing one correct charset,
> you're very naive. The problems are much hairier than that, and much
> more detailed.

Of course I understand Unicode: I am designing a compliant text editor
widget as I write this.  As far as C++ is concerned it really is that
simple.

/Flibble 

[toc] | [prev] | [next] | [standalone]

#44806

On 03.09.2016 21:10, Mr Flibble wrote:
> Richard Bos <raltbos@xs4all.nl> wrote:
>> Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> wrote:
>>
>>> On 29/08/2016 18:06, Lynn McGuire wrote:
>>>> "Unicode"
>>>> http://xkcd.com/1726/
>>>>
>>>> "Watching the Unicode people try to govern the infinite chaos of human
>>>> language with consistent technical standards is like watching highway
>>>> engineers try to steer a river using traffic signs."
>>>>
>>>> As usual, Randall Munroe nails it.
>>>
>>> Unicode is simple in C++: just use std::string and UTF-8.
>>
>> You... do not understand Unicode.
>>
>> If you think the problem is just one of choosing one correct charset,
>> you're very naive. The problems are much hairier than that, and much
>> more detailed.
>
> Of course I understand Unicode: I am designing a compliant text editor
> widget as I write this.  As far as C++ is concerned it really is that
> simple.

The common C and C++ standard libraries in Windows, as well as the 
Windows `char` based API functions, expect Windows ANSI encoding.

Which is not one single encoding but depends on the system.

That's a good reason for portable code to Just Say No to other encodings 
for plain `char` text, than the C++ execution character set.

In *nix-land it's easier, since the C++ execution character set is 
typically Unicode encoded as UTF-8, and since that can be part of the C 
locale.

In Windows it can't.


Cheers & hth.,

- Alf

[toc] | [prev] | [next] | [standalone]

#44810

On 4.09.2016 10:56, Alf P. Steinbach wrote:
> On 03.09.2016 21:10, Mr Flibble wrote:
>> Richard Bos <raltbos@xs4all.nl> wrote:
>>> Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> wrote:
>>>
>>>> On 29/08/2016 18:06, Lynn McGuire wrote:
>>>>> "Unicode"
>>>>> http://xkcd.com/1726/
>>>>>
>>>>> "Watching the Unicode people try to govern the infinite chaos of human
>>>>> language with consistent technical standards is like watching highway
>>>>> engineers try to steer a river using traffic signs."
>>>>>
>>>>> As usual, Randall Munroe nails it.
>>>>
>>>> Unicode is simple in C++: just use std::string and UTF-8.
>>>
>>> You... do not understand Unicode.
>>>
>>> If you think the problem is just one of choosing one correct charset,
>>> you're very naive. The problems are much hairier than that, and much
>>> more detailed.
>>
>> Of course I understand Unicode: I am designing a compliant text editor
>> widget as I write this.  As far as C++ is concerned it really is that
>> simple.
>
> The common C and C++ standard libraries in Windows, as well as the
> Windows `char` based API functions, expect Windows ANSI encoding.

That's why you never use Windows 'char' based API functions.

> Which is not one single encoding but depends on the system.
>
> That's a good reason for portable code to Just Say No to other encodings
> for plain `char` text, than the C++ execution character set.
>
> In *nix-land it's easier, since the C++ execution character set is
> typically Unicode encoded as UTF-8, and since that can be part of the C
> locale.
>
> In Windows it can't.

No, you can perfectly well use UTF-8 based strings in Windows, I have 
done this for 15 years. You just define a couple of convenience 
functions for easy conversion to-from UTF-16 at the application border 
(or to-from any other encoding when calling a library which expects 
another encoding).

Cheers
Paavo

[toc] | [prev] | [next] | [standalone]

#44858

On 04.09.2016 13:37, Paavo Helde wrote:
> On 4.09.2016 10:56, Alf P. Steinbach wrote:
>>
>> In *nix-land it's easier, since the C++ execution character set is
>> typically Unicode encoded as UTF-8, and since that can be part of the C
>> locale.
>>
>> In Windows it can't.
>
> No, you can perfectly well use UTF-8 based strings in Windows,

It can't be part of the C locale.

That's like, period.


> I have done this for 15 years.

No you haven't.

You may have used UTF-8 in your programs in Windows, as you can on any 
platform, for 15 years. That's something else. It's not practical when 
quality is important, but that's a very different discussion.

You have not used UTF-8 C locales in Windows.


> You just define a couple of convenience
> functions for easy conversion to-from UTF-16 at the application border
> (or to-from any other encoding when calling a library which expects
> another encoding).

Well, I'm sorry to say this, but that code probably have some severe 
limitations on filenames etc.

Generally, code originating in the *nix world, or based on such 
software, fares badly in Windows. For example, filenames and paths for 
Open CV library. Not to mention for the Code Blocks IDE.

I wouldn't mind the limitation to pure ASCII so much if they just could 
add support for spaces, for Pete's sake. It's like something from the 
dark ages. Or before.


Cheers,

- Alf

[toc] | [prev] | [next] | [standalone]

#44862

On 5.09.2016 13:53, Alf P. Steinbach wrote:
> On 04.09.2016 13:37, Paavo Helde wrote:
>> On 4.09.2016 10:56, Alf P. Steinbach wrote:
>>>
>>> In *nix-land it's easier, since the C++ execution character set is
>>> typically Unicode encoded as UTF-8, and since that can be part of the C
>>> locale.
>>>
>>> In Windows it can't.
>>
>> No, you can perfectly well use UTF-8 based strings in Windows,
>
> It can't be part of the C locale.
>
> That's like, period.
>
>
>> I have done this for 15 years.
>
> No you haven't.
>
> You may have used UTF-8 in your programs in Windows, as you can on any
> platform, for 15 years. That's something else. It's not practical when
> quality is important, but that's a very different discussion.
>
> You have not used UTF-8 C locales in Windows.

I have not claimed that. And it would not work anyway because as I wrote 
everything is converted to UTF-16 at the app border by the app itself, 
so using a UTF-8 locale would be very wrong.

All this locale stuff is pretty evil anyway (as every other hidden 
global state). Our code always uses either explicit "C" locale via 
functions like strtod_l() or enforces "C" locale otherwise. By using 
locale-dependent functions the functionality of the program would become 
unpredictable and very hard to maintain. The clear negative example is 
Excel where you can never know what punctuation you must enter in a 
formula because the syntax is heavily locale-dependent, and copy-pasting 
a piece of CSV most often does not work correctly.

Using locales might be useful only in the immediate user interface, and 
even there it is often more misleading or irritating than useful (e.g. 
random pieces of local language in long-formatted dates, in an otherwise 
correct English text). If you call that 'quality', then I humbly beg to 
differ.

Fortunately enough, most of my work is not related to UI and any 
locale-sensitivity is considered as a bug, not a feature.

>
>> You just define a couple of convenience
>> functions for easy conversion to-from UTF-16 at the application border
>> (or to-from any other encoding when calling a library which expects
>> another encoding).
>
> Well, I'm sorry to say this, but that code probably have some severe
> limitations on filenames etc.

Care to bring some examples? As far as I know, there is 1:1 translation 
from UTF-8 to UTF-16 used by Windows SDK file functions. It's true that 
not all Unix world filenames like those containing colons or backslashes 
are supported by Windows, but this has nothing to do with UTF-8.

>
> Generally, code originating in the *nix world, or based on such
> software, fares badly in Windows. For example, filenames and paths for
> Open CV library. Not to mention for the Code Blocks IDE.
>
> I wouldn't mind the limitation to pure ASCII so much if they just could
> add support for spaces, for Pete's sake. It's like something from the
> dark ages. Or before.

Sorry, here you lost me. Windows has supported spaces in filenames for 
20 years already. Probably you are talking about something else.

Cheers
Paavo

[toc] | [prev] | [next] | [standalone]

#44864

On 05.09.2016 14:30, Paavo Helde wrote:
> On 5.09.2016 13:53, Alf P. Steinbach wrote:
>> On 04.09.2016 13:37, Paavo Helde wrote:
>>> On 4.09.2016 10:56, Alf P. Steinbach wrote:
>>>>
>>>> In *nix-land it's easier, since the C++ execution character set is
>>>> typically Unicode encoded as UTF-8, and since that can be part of the C
>>>> locale.
>>>>
>>>> In Windows it can't.
>>>
>>> No, you can perfectly well use UTF-8 based strings in Windows,
>>
>> It can't be part of the C locale.
>>
>> That's like, period.
>>
>>
>>> I have done this for 15 years.
>>
>> No you haven't.
>>
>> You may have used UTF-8 in your programs in Windows, as you can on any
>> platform, for 15 years. That's something else. It's not practical when
>> quality is important, but that's a very different discussion.
>>
>> You have not used UTF-8 C locales in Windows.
>
> I have not claimed that.

Well, in context your "no" indicated a claim in that direction.

Except for context the "no" could mean anything, really.

It seems now that you meant a much wider context than just the preceding 
paragraph or two, but that was not obvious to me, sorry.


> And it would not work anyway because as I wrote
> everything is converted to UTF-16 at the app border by the app itself,
> so using a UTF-8 locale would be very wrong.
>
> All this locale stuff is pretty evil anyway (as every other hidden
> global state). Our code always uses either explicit "C" locale via
> functions like strtod_l() or enforces "C" locale otherwise. By using
> locale-dependent functions the functionality of the program would become
> unpredictable and very hard to maintain.

Oh, mostly I agree with that sentiment.

Still,

• A host of C standard library functions are locale dependent.

• Even C++ wide streams are locale dependent. In particular they don't 
work in *nix-land unless you set the locale, e.g. `setlocale( LC_ALL, "" 
)` in `main`. That's because they translate to/from external.

• Even string comparison, uppercasing and lowercasing in Unicode is to 
some degree locale dependent. However, I've always ignored that 
subtlety. I think, to the degree it's a problem one should just leave it 
to the users to deal properly with it, e.g. by avoiding troublesome 
¹Turkish or whatever characters.


> The clear negative example is
> Excel where you can never know what punctuation you must enter in a
> formula because the syntax is heavily locale-dependent, and copy-pasting
> a piece of CSV most often does not work correctly.

Yeah, agree, the Excel team/managers were insane.

They even at some point tried to provide national language versions of 
the scripting language, making not just CSV files but also scripts, 
effectively non-portable.


> Using locales might be useful only in the immediate user interface, and
> even there it is often more misleading or irritating than useful (e.g.
> random pieces of local language in long-formatted dates, in an otherwise
> correct English text). If you call that 'quality', then I humbly beg to
> differ.

By [lack of] "quality" I was referring to e.g. the common restriction to 
plain ASCII for *nix based code such as Open CV, lack of reliable 
support for spaces in paths such as in Code Blocks, lack of support for 
the backspace key such as in an installer for Qt at one point (I didn't 
know whether to laugh or cry), and so on, and on, and forth.


> Fortunately enough, most of my work is not related to UI and any
> locale-sensitivity is considered as a bug, not a feature.
>
>>
>>> You just define a couple of convenience
>>> functions for easy conversion to-from UTF-16 at the application border
>>> (or to-from any other encoding when calling a library which expects
>>> another encoding).
>>
>> Well, I'm sorry to say this, but that code probably have some severe
>> limitations on filenames etc.
>
> Care to bring some examples?

std::ofstream f( utf8_file_path ); //  OK in *nix, ungood in Windows.



> As far as I know, there is 1:1 translation
> from UTF-8 to UTF-16 used by Windows SDK file functions. It's true that
> not all Unix world filenames like those containing colons or backslashes
> are supported by Windows, but this has nothing to do with UTF-8.

Windows uses the colon as a stream name separator.

<example>
[c:\my\forums\clc++\036]
 > echo bah >knurre.txt:comment

[c:\my\forums\clc++\036]
 > dir | find "knurre.txt"
05.09.2016  16:07                 0 knurre.txt

[c:\my\forums\clc++\036]
 > more <knurre.txt:comment
bah

[c:\my\forums\clc++\036]
 > _
</example>

General *nix filenames /can/ be handled in an app, but the easiest then 
is to instead is to use whatever they now have instead of Interix 
subsystem for Unix.


>> Generally, code originating in the *nix world, or based on such
>> software, fares badly in Windows. For example, filenames and paths for
>> Open CV library. Not to mention for the Code Blocks IDE.
>>
>> I wouldn't mind the limitation to pure ASCII so much if they just could
>> add support for spaces, for Pete's sake. It's like something from the
>> dark ages. Or before.
>
> Sorry, here you lost me. Windows has supported spaces in filenames for
> 20 years already. Probably you are talking about something else.

Context. I'm talking about, quote from immediately preceding paragraph, 
"code originating in the *nix world, or based on such software, fares 
badly in Windows. For example, filenames and paths for Open CV library. 
Not to mention for the Code Blocks IDE."

E.g. last I tried it, Code Blocks still couldn't handle spaces in paths 
in general.

I think using UTF-8 encoding internally in a program in Windows can be 
OK for some special application domains, when that reduces porting work. 
Presumably the code you've been working on has been such.


Cheers!,

- Alf

Notes:
¹ See e.g. <url: http://www.i18nguy.com/unicode/turkish-i18n.html>

[toc] | [prev] | [next] | [standalone]

#44869

On 5.09.2016 17:13, Alf P. Steinbach wrote:
> On 05.09.2016 14:30, Paavo Helde wrote:
>> On 5.09.2016 13:53, Alf P. Steinbach wrote:
>>> On 04.09.2016 13:37, Paavo Helde wrote:
>>>> You just define a couple of convenience
>>>> functions for easy conversion to-from UTF-16 at the application border
>>>> (or to-from any other encoding when calling a library which expects
>>>> another encoding).
>>>
>>> Well, I'm sorry to say this, but that code probably have some severe
>>> limitations on filenames etc.
>>
>> Care to bring some examples?
>
> std::ofstream f( utf8_file_path ); //  OK in *nix, ungood in Windows.

Just tested this in Windows and it created the expected file ('(c) 
abc.txt') nicely.

#include <string>
#include <iostream>
#include <fstream>

#ifdef _WIN32
typedef std::wstring ExternalString;
#else
typedef std::string ExternalString;
#endif

/// Convert from UTF-8 to external platform-specific encoding
ExternalString ToExt(const std::string& utf8string);

int main() {

	std::string filename_utf8 = "C:\\temp\\\xC2\xA9 abc.txt";

	std::ofstream f(ToExt(filename_utf8).c_str());

}

#ifdef _WIN32
#include <Windows.h>

ExternalString ToExt(const std::string& utf8string) {
	ExternalString result(utf8string.length(), 0);
	if (!utf8string.empty()) {
		int n = MultiByteToWideChar(CP_UTF8, 0, utf8string.data(), 
utf8string.length(), &result[0], result.length());
		// Error check omitted for brevity
		result.resize(n);
	}
	return result;
}

#else

ExternalString ToExt(const std::string& utf8string) {
	return utf8string;
}
#endif

[toc] | [prev] | [next] | [standalone]

#44882

On 05.09.2016 21:03, Paavo Helde wrote:
> On 5.09.2016 17:13, Alf P. Steinbach wrote:
>> On 05.09.2016 14:30, Paavo Helde wrote:
>>> On 5.09.2016 13:53, Alf P. Steinbach wrote:
>>>> On 04.09.2016 13:37, Paavo Helde wrote:
>>>>> You just define a couple of convenience
>>>>> functions for easy conversion to-from UTF-16 at the application border
>>>>> (or to-from any other encoding when calling a library which expects
>>>>> another encoding).
>>>>
>>>> Well, I'm sorry to say this, but that code probably have some severe
>>>> limitations on filenames etc.
>>>
>>> Care to bring some examples?
>>
>> std::ofstream f( utf8_file_path ); //  OK in *nix, ungood in Windows.
>
> Just tested this in Windows and it created the expected
> file ('(c) abc.txt') nicely.

No, you tested with an UTF-16 file path, of type std::wstring.

That wide string std::ofstream constructor is a Microsoft extension, 
although it will effectively be required for Windows implementations of 
C++ by the filesystem library of C++17.

As a result the code fails to compile with e.g. g++ 5.1.0.

     • • •

Most (all?) g++ distros for Windows default to UTF-8 as the C++ 
execution character set. So ideally one could just use the narrow string 
directly for g++, i.e., a little compiler dependency, that's not a large 
cost. Unfortunately in Windows the default g++ standard library 
implementations expect Windows ANSI, i.e. incompatible with the 
compiler's C++ execution character set, and there's no way to change the 
locale to an UTF-8 one.

     • • •

The system dependent ExternalString string in your code, wide in Windows 
and narrow elsewhere, is IMO a good idea, though. I've advocated it for 
long, although I call it System_string. With proper support it can be 
used to handle text /internally/ in a program, avoiding all the problems 
with incompatibility between the internal representation and the 
standard library, OS API etc., and avoiding most of the gross 
inefficiencies, like dynamic allocations, of conversions to/from.

That proper support isn't trivial, though.

It includes e.g. a fix for the g++ problem noted above.


Cheers & hth.,

- Alf

[toc] | [prev] | [next] | [standalone]

#44883

"Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> writes:
[...]
> Most (all?) g++ distros for Windows default to UTF-8 as the C++ 
> execution character set. So ideally one could just use the narrow string 
> directly for g++, i.e., a little compiler dependency, that's not a large 
> cost. Unfortunately in Windows the default g++ standard library 
> implementations expect Windows ANSI, i.e. incompatible with the 
> compiler's C++ execution character set, and there's no way to change the 
> locale to an UTF-8 one.

"ANSI" is an incorrect name for the Windows-1252 8-bit character set
(and related code pages).  Even Microsoft no longer calls it that.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#44887

On 06.09.2016 00:35, Keith Thompson wrote:
> "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> writes:
> [...]
>> Most (all?) g++ distros for Windows default to UTF-8 as the C++
>> execution character set. So ideally one could just use the narrow string
>> directly for g++, i.e., a little compiler dependency, that's not a large
>> cost. Unfortunately in Windows the default g++ standard library
>> implementations expect Windows ANSI, i.e. incompatible with the
>> compiler's C++ execution character set, and there's no way to change the
>> locale to an UTF-8 one.
>
> "ANSI" is an incorrect name for the Windows-1252 8-bit character set
> (and related code pages).  Even Microsoft no longer calls it that.

“Windows ANSI” is not a name for a specific character set.

It's the codepage specified by the ¹GetACP API function, which, not very 
well documented, is the codepage specified in the <registry: 
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage> key, 
item name ACP, which is short for “ANSI code page”.

The information you have about deprecation of the name “ANSI” may well 
be correct. Microsoft has, after all, deprecated nearly all of the 
original string functions of the C library. That is, they believed they 
had: but it was, just like an attempted deprecation of the name used in 
their APIs, documentation, books etc., just utter nonsense. :)


Cheers & hth.,

- Alf

¹ Documented at <url: 
https://msdn.microsoft.com/en-us/library/windows/desktop/dd318070(v=vs.85).aspx>

[toc] | [prev] | [next] | [standalone]

#44888

"Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> writes:
> On 06.09.2016 00:35, Keith Thompson wrote:
[...]
>> "ANSI" is an incorrect name for the Windows-1252 8-bit character set
>> (and related code pages).  Even Microsoft no longer calls it that.
>
> “Windows ANSI” is not a name for a specific character set.
>
> It's the codepage specified by the ¹GetACP API function, which, not very 
> well documented, is the codepage specified in the <registry: 
> HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage> key, 
> item name ACP, which is short for “ANSI code page”.

ACP may originally have been an abbreviation of "ANSI Code Page", and
apparently it's still used as a key name for backward compatibility
(which sounds reasonable).

My point is that "Windows ANSI" is a misnomer.  The character sets in
question were intended to be an ANSI standard(s), but they were never
approved.  The collection of code pages sometimes referred to as
"Windows ANSI" have nothing to do with the American National Standards
Institute.

> The information you have about deprecation of the name “ANSI” may well 
> be correct. Microsoft has, after all, deprecated nearly all of the 
> original string functions of the C library. That is, they believed they 
> had: but it was, just like an attempted deprecation of the name used in 
> their APIs, documentation, books etc., just utter nonsense. :)

That's not quite the same.  Microsoft's attempt to deprecate the
standard string functions is absurd, since they're still in the ISO C
standard.  Deprecating the name "ANSI" for certain Windows-specific
character sets, on the other hand, is entirely reasonable.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#44889

On 06.09.2016 01:17, Keith Thompson wrote:
> "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> writes:
>> On 06.09.2016 00:35, Keith Thompson wrote:
> [...]
>>> "ANSI" is an incorrect name for the Windows-1252 8-bit character set
>>> (and related code pages).  Even Microsoft no longer calls it that.
>>
>> “Windows ANSI” is not a name for a specific character set.
>>
>> It's the codepage specified by the ¹GetACP API function, which, not very
>> well documented, is the codepage specified in the <registry:
>> HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage> key,
>> item name ACP, which is short for “ANSI code page”.
>
> ACP may originally have been an abbreviation of "ANSI Code Page", and
> apparently it's still used as a key name for backward compatibility
> (which sounds reasonable).
>
> My point is that "Windows ANSI" is a misnomer.

In that, you're right.

But I have not written or implied it's not.

Please provide a quote to back up that implied assertion.


>  The character sets in question were intended to be an ANSI standard(s),

No.

Windows ANSI Western, codepage 1252, was based on a draft of ISO 
Latin-1, then, as I recall, developed in cooperation between the ANSI 
institute in America, and its French counterpart.

It was not intended to an ANSI standard: it was possibly intended to be 
one-way compatible with the ANSI standard ISO Latin-1, since it is a 
strict superset.

Regarding the other Windows ANSI character sets they include e.g. Shift 
JIS, which has almost no relation to Windows ANSI Western.

I believe some of them are ANSI and ISO standards, but maybe not.


> but they were never approved.

Please provide evidence that they were submitted for standardization.


>  The collection of code pages sometimes referred to as
> "Windows ANSI" have nothing to do with the American National Standards
> Institute.

In that, you're right.

But I have not written or implied the opposite.

I think you're just trying to associate with me with the silly notions 
that you APPEAR to be arguing against.

If so then that's not very honest argumentation.


>> The information you have about deprecation of the name “ANSI” may well
>> be correct. Microsoft has, after all, deprecated nearly all of the
>> original string functions of the C library. That is, they believed they
>> had: but it was, just like an attempted deprecation of the name used in
>> their APIs, documentation, books etc., just utter nonsense. :)
>
> That's not quite the same.

Oh, it is. Microsoft do things that are technically nonsense, apparently 
focusing on vendor lock-in (giving rise to the ¹EEE acronym: Embrace, 
Extend and Extinguish) and on social issues such as, in this case (if 
you're right about it), in-group terminology. In the case of the 
attempted deprecation of C library functions it was linked to an 
in-group terminology and an in-group set of annotation and checking 
tools, that maybe have some value in C but just are a hassle in C++.


>  Microsoft's attempt to deprecate the
> standard string functions is absurd, since they're still in the ISO C
> standard.  Deprecating the name "ANSI" for certain Windows-specific
> character sets, on the other hand, is entirely reasonable.

No, that's also incorrect.

E.g. check the documentation of Windows' GetACP function, which I linked to.

It's not “entirely resonable” to change a term that's hardwired into API 
functions, documentation, books, ²Wikipedia, etc. ad nauseam.


Cheers & hth.,

- Alf

¹ https://en.wikipedia.org/wiki/Embrace,_extend_and_extinguish
² https://en.wikipedia.org/wiki/Windows_code_page#ANSI_code_page

[toc] | [prev] | [next] | [standalone]

#44890

"Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> writes:
> On 06.09.2016 01:17, Keith Thompson wrote:
[...]
>> My point is that "Windows ANSI" is a misnomer.
>
> In that, you're right.
>
> But I have not written or implied it's not.

You use the phrase without pointing out that it's incorrect.

> Please provide a quote to back up that implied assertion.

From upthread:

    Unfortunately in Windows the default g++ standard library
    implementations expect Windows ANSI, i.e. incompatible with the
    compiler's C++ execution character set, and there's no way to
    change the locale to an UTF-8 one.

>>  The character sets in question were intended to be an ANSI standard(s),
>
> No.
>
> Windows ANSI Western, codepage 1252, was based on a draft of ISO 
> Latin-1, then, as I recall, developed in cooperation between the ANSI 
> institute in America, and its French counterpart.
>
> It was not intended to an ANSI standard: it was possibly intended to be 
> one-way compatible with the ANSI standard ISO Latin-1, since it is a 
> strict superset.
>
> Regarding the other Windows ANSI character sets they include e.g. Shift 
> JIS, which has almost no relation to Windows ANSI Western.
>
> I believe some of them are ANSI and ISO standards, but maybe not.
>
>
>> but they were never approved.
>
> Please provide evidence that they were submitted for standardization.

Quoting https://en.wikipedia.org/wiki/Windows_code_page#ANSI_code_page :

    ANSI Windows code pages, and especially the code page 1252,
    were called that way since they were purportedly based on drafts
    submitted or intended for ANSI. However, ANSI and ISO have not
    standardized any of these code pages.

My statement seems to have been imprecise.

>>  The collection of code pages sometimes referred to as
>> "Windows ANSI" have nothing to do with the American National Standards
>> Institute.
>
> In that, you're right.
>
> But I have not written or implied the opposite.
>
> I think you're just trying to associate with me with the silly notions 
> that you APPEAR to be arguing against.
>
> If so then that's not very honest argumentation.

I think you're overreacting to my criticism.  You referred to "Windows
ANSI".  There's really no such thing, though it's a common misuse.

[snip]

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#44891

On 06.09.2016 02:16, Keith Thompson wrote:
> "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> writes:
>> On 06.09.2016 01:17, Keith Thompson wrote:
> [...]
>>> My point is that "Windows ANSI" is a misnomer.
>>
>> In that, you're right.
>>
>> But I have not written or implied it's not.
>
> You use the phrase without pointing out that it's incorrect.

No, the term “Windows ANSI” is not incorrect at all: it's the name of 
this concept. You can read all about it in Microsoft's documentation 
(which I linked to) and in Wikipedia (which I linked to). It's a 
misnomer when it's (foolishly) treated as a descriptive term, and it is 
the name of this concept.



>> Please provide a quote to back up that implied assertion.
>
> From upthread:
>
>     Unfortunately in Windows the default g++ standard library
>     implementations expect Windows ANSI, i.e. incompatible with the
>     compiler's C++ execution character set, and there's no way to
>     change the locale to an UTF-8 one.

So, where in that quote is the argument that you're arguing against?

Again, I perceive your argumentation as merely a way to throw out 
negative associations.

You'd have to work pretty hard to convince me otherwise.


>>>  The character sets in question were intended to be an ANSI standard(s),
>>
>> No.
>>
>> Windows ANSI Western, codepage 1252, was based on a draft of ISO
>> Latin-1, then, as I recall, developed in cooperation between the ANSI
>> institute in America, and its French counterpart.
>>
>> It was not intended to an ANSI standard: it was possibly intended to be
>> one-way compatible with the ANSI standard ISO Latin-1, since it is a
>> strict superset.
>>
>> Regarding the other Windows ANSI character sets they include e.g. Shift
>> JIS, which has almost no relation to Windows ANSI Western.
>>
>> I believe some of them are ANSI and ISO standards, but maybe not.
>>
>>> but they were never approved.
>>
>> Please provide evidence that they were submitted for standardization.
>
> Quoting https://en.wikipedia.org/wiki/Windows_code_page#ANSI_code_page :
>
>     ANSI Windows code pages, and especially the code page 1252,
>     were called that way since they were purportedly based on drafts
>     submitted or intended for ANSI. However, ANSI and ISO have not
>     standardized any of these code pages.

I think “any of these code pages” is an unwarranted and incorrect 
generalization in that quote.

But even if the Wikipedia text here were correct, which it doesn't seem 
to be, it would not then back up your claim that the code pages were 
“never approved” for standardization. For in order for approval to be a 
possibility they'd have to be submittedm. The quote does not say they've 
been submitted.


> My statement seems to have been imprecise.
>
>>>  The collection of code pages sometimes referred to as
>>> "Windows ANSI" have nothing to do with the American National Standards
>>> Institute.
>>
>> In that, you're right.
>>
>> But I have not written or implied the opposite.
>>
>> I think you're just trying to associate with me with the silly notions
>> that you APPEAR to be arguing against.
>>
>> If so then that's not very honest argumentation.
>
> I think you're overreacting to my criticism.  You referred to "Windows
> ANSI".  There's really no such thing, though it's a common misuse.

It's not a common misuse: it's the name of this concept. Again, you can 
read all about it in Microsoft's documentation (which I linked to) and 
in Wikipedia (which I linked to). It's misnomer, and it is the name. 
Lots of names are misnomers. This name is one, and still, just like 
other misnomer names, it's the name; what you use to refer to the thing.


Cheers!,

- Alf

[toc] | [prev] | [next] | [standalone]

#44884

On 05/09/2016 22:54, Alf P. Steinbach wrote:
> On 05.09.2016 21:03, Paavo Helde wrote:
>> On 5.09.2016 17:13, Alf P. Steinbach wrote:
>>> On 05.09.2016 14:30, Paavo Helde wrote:
>>>> On 5.09.2016 13:53, Alf P. Steinbach wrote:
>>>>> On 04.09.2016 13:37, Paavo Helde wrote:
>>>>>> You just define a couple of convenience
>>>>>> functions for easy conversion to-from UTF-16 at the application
>>>>>> border
>>>>>> (or to-from any other encoding when calling a library which expects
>>>>>> another encoding).
>>>>>
>>>>> Well, I'm sorry to say this, but that code probably have some severe
>>>>> limitations on filenames etc.
>>>>
>>>> Care to bring some examples?
>>>
>>> std::ofstream f( utf8_file_path ); //  OK in *nix, ungood in Windows.
>>
>> Just tested this in Windows and it created the expected
>> file ('(c) abc.txt') nicely.
>
> No, you tested with an UTF-16 file path, of type std::wstring.
>
> That wide string std::ofstream constructor is a Microsoft extension,
> although it will effectively be required for Windows implementations of
> C++ by the filesystem library of C++17.
>
> As a result the code fails to compile with e.g. g++ 5.1.0.
>
>     • • •
>
> Most (all?) g++ distros for Windows default to UTF-8 as the C++
> execution character set. So ideally one could just use the narrow string
> directly for g++, i.e., a little compiler dependency, that's not a large
> cost. Unfortunately in Windows the default g++ standard library
> implementations expect Windows ANSI, i.e. incompatible with the
> compiler's C++ execution character set, and there's no way to change the
> locale to an UTF-8 one.
>
>     • • •
>
> The system dependent ExternalString string in your code, wide in Windows
> and narrow elsewhere, is IMO a good idea, though. I've advocated it for
> long, although I call it System_string. With proper support it can be
> used to handle text /internally/ in a program, avoiding all the problems
> with incompatibility between the internal representation and the
> standard library, OS API etc., and avoiding most of the gross
> inefficiencies, like dynamic allocations, of conversions to/from.
>
> That proper support isn't trivial, though.
>
> It includes e.g. a fix for the g++ problem noted above.

boost.filesystem

/Flibble

[toc] | [prev] | [next] | [standalone]

#44834

On 04/09/2016 08:56, Alf P. Steinbach wrote:
> In *nix-land it's easier, since the C++ execution character set is
> typically Unicode encoded as UTF-8, and since that can be part of the C
> locale.
>
> In Windows it can't.

I don't recall the API off the top of my head, but I do recall that you 
can set your character set to UTF-8 in Windows.

Doubtless not in a portable manner... but then AIUI setting it to 
anything else can't be done on Linux.

Andy

[toc] | [prev] | [next] | [standalone]

#44840

On 4.09.2016 23:37, Vir Campestris wrote:
> On 04/09/2016 08:56, Alf P. Steinbach wrote:
>> In *nix-land it's easier, since the C++ execution character set is
>> typically Unicode encoded as UTF-8, and since that can be part of the C
>> locale.
>>
>> In Windows it can't.
>
> I don't recall the API off the top of my head, but I do recall that you
> can set your character set to UTF-8 in Windows.

Sure you can set it ... but the machine won't boot up again after 
restart. At least this was my experience when I last tried that.

To be honest, this was with Win7 several years ago. Maybe they have 
fixed this meanwhile, if so, it would mean a major breakthrough towards 
portable software. But somehow I doubt it.

> Doubtless not in a portable manner... but then AIUI setting it to
> anything else can't be done on Linux.

If this is so, then probably this is because setting it to anything else 
has no point, so no software has been tested with it.

Cheers
Paavo

[toc] | [prev] | [next] | [standalone]

Page 3 of 4 — ← Prev page 1 2 [3] 4  Next page →

Back to top | Article view | comp.lang.c++

csiph-web