Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #6354 > unrolled thread

UTF-8 and strings

Back to article view | Back to comp.lang.c++

Contents

  UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-07 08:08 -0700
    Re: UTF-8 and strings Asger-P <junk@asger-p.dk> - 2011-06-07 17:26 +0200
      Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-08 17:58 -0700
        Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-08 18:21 -0700
          Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-09 06:38 -0700
    Re: UTF-8 and strings "Balog Pal" <pasa@lib.hu> - 2011-06-07 17:35 +0200
      Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-08 18:08 -0700
    Re: UTF-8 and strings Jeff Flinn <TriumphSprint2000@hotmail.com> - 2011-06-07 12:55 -0400
      Re: UTF-8 and strings Paavo Helde <myfirstname@osa.pri.ee> - 2011-06-07 15:29 -0500
        Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-08 07:15 -0500
          Re: UTF-8 and strings yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-06-08 15:45 +0000
            Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-08 12:58 -0500
              Re: UTF-8 and strings yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-06-09 09:18 +0000
                Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-09 07:36 -0500
                  Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-10 13:18 -0700
                    Re: UTF-8 and strings Ruben Safir <mrbrklyn@panix.com> - 2011-06-11 00:29 +0000
      Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-10 12:58 -0700
    Re: UTF-8 and strings diamondback <christopher.rau@gmail.com> - 2011-06-07 10:57 -0700
      Re: UTF-8 and strings Jorgen Grahn <grahn+nntp@snipabacken.se> - 2011-06-08 06:38 +0000
    Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-07 13:30 -0700
      Re: UTF-8 and strings Jorgen Grahn <grahn+nntp@snipabacken.se> - 2011-06-08 06:52 +0000
        Re: UTF-8 and strings Marc <marc.glisse@gmail.com> - 2011-06-08 08:38 +0000
          Re: UTF-8 and strings Miles Bader <miles@gnu.org> - 2011-06-08 18:02 +0900
            Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-08 02:05 -0700
              Re: UTF-8 and strings Miles Bader <miles@gnu.org> - 2011-06-08 18:11 +0900
            Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-08 07:22 -0500
          Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-08 02:03 -0700
            Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-08 02:06 -0700
          Re: UTF-8 and strings tm <thomas.mertes@gmx.at> - 2011-06-08 02:59 -0700
          Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-08 18:20 -0700
        Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-08 01:56 -0700
        Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-08 07:19 -0500
          Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-10 13:12 -0700
            Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-10 18:34 -0500
              Re: UTF-8 and strings Miles Bader <miles@gnu.org> - 2011-06-11 13:12 +0900
              Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-12 20:52 -0700
                Re: UTF-8 and strings ruben safir <ruben@mrbrklyn.com> - 2011-06-13 03:57 +0000
                  Re: UTF-8 and strings Asger-P <junk@asger-p.dk> - 2011-06-13 10:51 +0200
      Re: UTF-8 and strings Nobody <nobody@nowhere.com> - 2011-06-08 08:01 +0100
    Re: UTF-8 and strings Nobody <nobody@nowhere.com> - 2011-06-07 21:35 +0100
      Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-08 01:56 -0700
        Re: UTF-8 and strings Nobody <nobody@nowhere.com> - 2011-06-09 03:22 +0100
    Re: UTF-8 and strings yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-06-08 15:55 +0000
      Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-08 17:55 -0700
        Re: UTF-8 and strings yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-06-09 10:02 +0000
          Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-09 08:41 -0500
            Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-12 20:57 -0700
              Re: UTF-8 and strings ruben safir <ruben@mrbrklyn.com> - 2011-06-13 04:46 +0000
                Re: UTF-8 and strings Asger-P <junk@asger-p.dk> - 2011-06-13 11:58 +0200
          Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-09 06:52 -0700

Page 1 of 3  [1] 2 3  Next page →

#6354 — UTF-8 and strings

I may soon be working on a code base that is undergoing
"internationalization", and what is already done on low levels of
operating system interaction is using UTF-8 in char-based strings.

Clearly, new code needs to use Unicode, or _some_ way of representing
a lot more than 256 characters.  UTF-8 and Unicode has merit as an
encoding and transmission format, and I'm certainly not against it.

But std::string models a sequence of *characters*, and that doesn't
fit with multi-byte variable-length character encodings of any kind.
Furthermore, one character won't fit in a char.

Before I get in too deep with my thoughts, I want to see what others
are doing.  Is there any existing information on this topic?  Is there
a better place to discuss it?

My initial thoughts is that most of the time the code simply hangs
onto the string data and doesn't manipulate it, so "funny" contents
won't matter much.  Furthermore, UTF-8 avoids (by design) many of the
problems found with multi-byte encodings, so a naive handling might
work "well enough" for common tasks.  However, I should catalog in
detail just what works and what the problems are.  For more general
manipulation of the string, we need functions like the C library's
mblen etc. and replacements for standard library routines whose
implementation doesn't handle UTF-8 as multi-byte character strings.

Or, is there some paradigm shift I should be aware of?

—John

[toc] | [next] | [standalone]

#6357

Hi John

I am by no means a C++ expert, I use C++Builder which is full
unicode now, but I can see that there is a wstring in STL so
my suggestion would be to translate all incoming and outgoing
data and the n internally use wchar_t and wstring.

You didn't mention if You were on a specific operating system,
but Windows work in wchar_t internally and I suppose unix, linux
and Mac does the same, I dont know though.


Best regards
Asger-P

[toc] | [prev] | [next] | [standalone]

#6462

On Jun 7, 10:26 am, Asger-P <j...@asger-p.dk> wrote:
> Hi John
>
> I am by no means a C++ expert, I use C++Builder which is full
> unicode now, but I can see that there is a wstring in STL so
> my suggestion would be to translate all incoming and outgoing
> data and the n internally use wchar_t and wstring.
>
> You didn't mention if You were on a specific operating system,
> but Windows work in wchar_t internally and I suppose unix, linux
> and Mac does the same, I dont know though.
>
> Best regards
> Asger-P

The question isn't "are there better ways to handle things than using
UTF-8 internally."
My question is, "*given* that this project uses UTF-8 internally, and
will do more of that as development continues, and the C++ standard
library doesn't have a class that models that concept cleanly, what
are good ways to deal with it?"
I'd love to solve the problem differently, and not care about memory
consumption either (just use 32-bit characters), but that's not where
I am on this.

—John

[toc] | [prev] | [next] | [standalone]

#6466

On Jun 8, 5:58 pm, "John M. Dlugosz" <uwqejpn...@snkmail.com> wrote:
> On Jun 7, 10:26 am, Asger-P <j...@asger-p.dk> wrote:
>
> > Hi John
>
> > I am by no means a C++ expert, I use C++Builder which is full
> > unicode now, but I can see that there is a wstring in STL so
> > my suggestion would be to translate all incoming and outgoing
> > data and the n internally use wchar_t and wstring.
>
> > You didn't mention if You were on a specific operating system,
> > but Windows work in wchar_t internally and I suppose unix, linux
> > and Mac does the same, I dont know though.
>
> > Best regards
> > Asger-P
>
> The question isn't "are there better ways to handle things than using
> UTF-8 internally."
> My question is, "*given* that this project uses UTF-8 internally, and
> will do more of that as development continues, and the C++ standard
> library doesn't have a class that models that concept cleanly, what
> are good ways to deal with it?"
> I'd love to solve the problem differently, and not care about memory
> consumption either (just use 32-bit characters), but that's not where
> I am on this.

Let me know if you figure that out. My own company uses ICU,
specifically a modified version of ICU to get rid of all of that
stupid virtual function call per character nonsense. It also
unfortunately uses basically UCS-2 strings. A lot of it was done
before UTF-32 and the newer Unicode standard came out.

If I were on a new project, I would like to use a pre-existing
library, but AFAIK no library handles it correctly out of the box.
Thus, I would personally want to write my own string class and divorce
it from the ill-specified and not portable C++ locale stuffs, though
I'd probably give it a std::stringstream equivalent.

What I would want from a good Unicode string class is:
- Well, 3 classes actually. I'd have utf8string, utf16string,
utf32string. I'd code it in such a way to ensure little code
duplication.
- Each one would have three different iterator types:
encoding_unit_iter, code_point_iter, and grapheme_cluster_iter.
- "begin" "end" et. al. get removed, and instead you have one begin/
end pair for each of the 3 iterator types (ex: begin_encoding_unit(),
end_encoding_unit()). encoding_unit_iter is a random access iterator,
and the rest are not. (Well, except for utf32 where code_point_iter is
also random access iterator. In the interests of easily changing
encodings in the program with a recompilation, I might have a separate
utf32string class for the expressed purpose of having its
code_point_iter be /not/ random access.) Of course, there would be
constructors to go from grapheme_cluster_iter to code_point_iter and
encoding_unit_iter, and code_point_iter to encoding_unit_iter. There
would be named conversion functions to force conversions the other way
(as they may not be valid). Note also that grapheme_cluster_iter may
be language and culture dependent (I forget?), and so the begin and
end functions for that may require a locale object (or whatever passes
for a locale object in my new crazy scheme).
- I would remove operator[], but have an explicit subscript function
named something like
    encoding_unit_t& utfstring::get_encoding_unit(size_t i);
    encoding_unit_t  utfstring::get_encoding_unit(size_t i) const;
- I would remove all other functions that use integer offsets. They
would be replaced with the following functions: ones that work in
terms of iterators that iterate over code points, and another that
works in terms of iterators of this string's encoding unit type. The
functions would have obvious names as such.
- I'm not sure if I would want to have a fully mutable interface, or
go read-only and make it thread-safe ala Java. It makes little sense
IMHO to allow mutable utfstrings in the general case. You can't
replace one character with another (except maybe in utf32 strings) as
the code points (or grapheme clusters!) may have different encoding
lengths.
- Stand alone functions that translate from one encoding to another.
Or actually stand alone objects for translation between encodings in
order to support stateful encodings (of which the UTFs are not, thank
god, but some encodings are).

My crazy scheme would probably make a lot of use of existing ICU
classes, such as for collation and translation. Especially it would
make use of their character and locale data files. (Assuming the
license would allow that. I don't know offhand. Gotta get the up to
date and machine readable character and locale data files from
somewhere.)

WARNING! Pardon my ranting. I've done little actual work with Unicode,
but what little work I have done have led me to these conclusions.
Take at your own risk.

[toc] | [prev] | [next] | [standalone]

#6493

On Jun 8, 8:21 pm, Joshua Maurice <joshuamaur...@gmail.com> wrote:
> What I would want from a good Unicode string class is: …

Thanks for your extensive notes.  I see your ideas are sometimes
different from what I've seen thus far of Glib::ustring, which is
designed for more plug-in replacement.

I would also consider how such a class plays with other classes that
use strings, such as the Boost and now TR2 filename class.

[toc] | [prev] | [next] | [standalone]

#6358

"John M. Dlugosz" <uwqejpnp92@snkmail.com>
>I may soon be working on a code base that is undergoing
>"internationalization", and what is already done on low levels of
>operating system interaction is using UTF-8 in char-based strings.

check out std::wstring, Glib::ustring for discussion. I'd bet boost also 
must have i18n related stuff. 

[toc] | [prev] | [next] | [standalone]

#6463

On Jun 7, 10:35 am, "Balog Pal" <p...@lib.hu> wrote:
> check out std::wstring, Glib::ustring for discussion. I'd bet boost also
> must have i18n related stuff.

Thanks for the pointer to Glib::ustring.  The description and
discussion sounds like exactly my concerns, and it is used by other
libraries already so it's probably nicely adopt-able.

—John

[toc] | [prev] | [next] | [standalone]

#6365

John M. Dlugosz wrote:
> I may soon be working on a code base that is undergoing
> "internationalization", and what is already done on low levels of
> operating system interaction is using UTF-8 in char-based strings.
> 
> Clearly, new code needs to use Unicode, or _some_ way of representing
> a lot more than 256 characters.  UTF-8 and Unicode has merit as an
> encoding and transmission format, and I'm certainly not against it.
> 
> But std::string models a sequence of *characters*, and that doesn't
> fit with multi-byte variable-length character encodings of any kind.
> Furthermore, one character won't fit in a char.
> 
> Before I get in too deep with my thoughts, I want to see what others
> are doing.  Is there any existing information on this topic?  Is there
> a better place to discuss it?
> 
> My initial thoughts is that most of the time the code simply hangs
> onto the string data and doesn't manipulate it, so "funny" contents
> won't matter much.  Furthermore, UTF-8 avoids (by design) many of the
> problems found with multi-byte encodings, so a naive handling might
> work "well enough" for common tasks.  However, I should catalog in
> detail just what works and what the problems are.  For more general
> manipulation of the string, we need functions like the C library's
> mblen etc. and replacements for standard library routines whose
> implementation doesn't handle UTF-8 as multi-byte character strings.
> 
> Or, is there some paradigm shift I should be aware of?

Check the boost mailing list. There's been some discussion recently, as 
well as a presentation at boostcon last month.

Jeff

[toc] | [prev] | [next] | [standalone]

#6376

Jeff Flinn <TriumphSprint2000@hotmail.com> wrote in news:isll5q$lmv$1
@dont-email.me:

> John M. Dlugosz wrote:
>> I may soon be working on a code base that is undergoing
>> "internationalization", and what is already done on low levels of
>> operating system interaction is using UTF-8 in char-based strings.
>> 
>> Clearly, new code needs to use Unicode, or _some_ way of representing
>> a lot more than 256 characters.  UTF-8 and Unicode has merit as an
>> encoding and transmission format, and I'm certainly not against it.
>> 
>> But std::string models a sequence of *characters*, and that doesn't
>> fit with multi-byte variable-length character encodings of any kind.
>> Furthermore, one character won't fit in a char.
>> 
>> Before I get in too deep with my thoughts, I want to see what others
>> are doing.  Is there any existing information on this topic?  Is there
>> a better place to discuss it?
>> 
>> My initial thoughts is that most of the time the code simply hangs
>> onto the string data and doesn't manipulate it, so "funny" contents
>> won't matter much.  Furthermore, UTF-8 avoids (by design) many of the
>> problems found with multi-byte encodings, so a naive handling might
>> work "well enough" for common tasks.  However, I should catalog in
>> detail just what works and what the problems are.  For more general
>> manipulation of the string, we need functions like the C library's
>> mblen etc. and replacements for standard library routines whose
>> implementation doesn't handle UTF-8 as multi-byte character strings.
>> 
>> Or, is there some paradigm shift I should be aware of?

By some reason I do not see your original post, so replying here. 

There is nothing wrong with UTF-8. It certainly fits well in std::string, 
as far as the strings encoded in UTF-8 are clearly understood and handled 
as such (as opposed to ones encoded in pure ASCII or in some other code 
page, for example). If there are different types of strings in play, 
Hungarian Notation may help to distinguish them. The only action which is 
compromised by the UTF-8 encoding is the extraction and/or replacement of 
a non-ASCII character in the string, but this seems to be not so frequent 
operation.

For length enquires there are already UTF-8 specific SDK functions 
available on some platforms (mbstowcs(), wcswidth()).

Potential problems appear when interfacing with external world, e.g. 
displaying a message box to the user. A consistent behavior would apply 
an appropriate transformation whenever the string crosses the application 
boundary - encoding to/from UTF-16 on Windows and doing nothing on a 
standard Linux installation, for example. 

hth
Paavo

[toc] | [prev] | [next] | [standalone]

#6416

Paavo Helde wrote:

> There is nothing wrong with UTF-8.

Unless one craves the simplicity of constant-width characters. 

[toc] | [prev] | [next] | [standalone]

#6426

In article <isnp5i$g9p$1@dont-email.me>, MikeP <mp011011@some.org> wrote:
>Paavo Helde wrote:
>
>> There is nothing wrong with UTF-8.
>
>Unless one craves the simplicity of constant-width characters. 
>

That pretty much forces you to UTF-32 then which is a massive waste. 

[toc] | [prev] | [next] | [standalone]

#6432

none (Yannick Tremblay) wrote:
> In article <isnp5i$g9p$1@dont-email.me>, MikeP <mp011011@some.org>
> wrote:
>> Paavo Helde wrote:
>>
>>> There is nothing wrong with UTF-8.
>>
>> Unless one craves the simplicity of constant-width characters.
>>
>
> That pretty much forces you to UTF-32 then which is a massive waste.

Or use UTF-16 as UCS-2, which sounds pretty reasonable still on Windows 
(?). 

[toc] | [prev] | [next] | [standalone]

#6475

In article <isod85$lf9$1@dont-email.me>, MikeP <mp011011@some.org> wrote:
>none (Yannick Tremblay) wrote:
>> In article <isnp5i$g9p$1@dont-email.me>, MikeP <mp011011@some.org>
>> wrote:
>>> Paavo Helde wrote:
>>>
>>>> There is nothing wrong with UTF-8.
>>>
>>> Unless one craves the simplicity of constant-width characters.
>>>
>>
>> That pretty much forces you to UTF-32 then which is a massive waste.
>
>Or use UTF-16 as UCS-2, which sounds pretty reasonable still on Windows 
>(?). 

Use real UCS-2: probably acceptable

Use UTF-16 as if it was UCS-2: Euh,  this is how bugs happens. 
- I want the simplicity of constant-width characters
- I choose UTF-16 but ignore the existence of surrogate pairs
- Ooops bang!

It's even on wikipedia (not a good reference but still)
"Because the most commonly used characters are all in the Basic
Multilingual Plane, handling of surrogate pairs is often not tested
thoroughly. This leads to persistent bugs, and potential security
holes, even in popular and well-reviewed application software"   

Yannick

[toc] | [prev] | [next] | [standalone]

#6488

none (Yannick Tremblay) wrote:
> In article <isod85$lf9$1@dont-email.me>, MikeP <mp011011@some.org>
> wrote:
>> none (Yannick Tremblay) wrote:
>>> In article <isnp5i$g9p$1@dont-email.me>, MikeP <mp011011@some.org>
>>> wrote:
>>>> Paavo Helde wrote:
>>>>
>>>>> There is nothing wrong with UTF-8.
>>>>
>>>> Unless one craves the simplicity of constant-width characters.
>>>>
>>>
>>> That pretty much forces you to UTF-32 then which is a massive waste.
>>
>> Or use UTF-16 as UCS-2, which sounds pretty reasonable still on
>> Windows (?).
>
> Use real UCS-2: probably acceptable
>
> Use UTF-16 as if it was UCS-2: Euh,  this is how bugs happens.
> - I want the simplicity of constant-width characters
> - I choose UTF-16 but ignore the existence of surrogate pairs
> - Ooops bang!

But isn't Windows (XP)'s "UTF-16" really UCS-2 and hence not a risk at 
all then? What I was getting at is whether Windows 7 has changed to REAL 
UTF-16. There is UTF-16 and then there is "UTF-16". What is REALLY under 
the hood, rather than what it is marketed as.

>
> It's even on wikipedia (not a good reference but still)
> "Because the most commonly used characters are all in the Basic
> Multilingual Plane, handling of surrogate pairs is often not tested
> thoroughly. This leads to persistent bugs, and potential security
> holes, even in popular and well-reviewed application software"
>
> Yannick 

[toc] | [prev] | [next] | [standalone]

#6582

On Jun 9, 7:36 am, "MikeP" <mp011...@some.org> wrote:
>
> But isn't Windows (XP)'s "UTF-16" really UCS-2 and hence not a risk at
> all then?

Originally, yes.  At some point it went to real UTF-16 awareness, at
least for rendering strings using fonts into windows, and converting
between code pages correctly.  This was around Windows-2000 or XP,
certainly well before Windows 7.

I think the file system is technically still USC-2 in that it doesn't
care what the code points mean and doesn't reject improper use of
surrogate pairs as errors.

[toc] | [prev] | [next] | [standalone]

#6598

John M. Dlugosz <uwqejpnp92@snkmail.com> wrote:
> On Jun 9, 7:36 am, "MikeP" <mp011...@some.org> wrote:
>>
>> But isn't Windows (XP)'s "UTF-16" really UCS-2 and hence not a risk at
>> all then?
> 
> Originally, yes.  At some point it went to real UTF-16 awareness, at
> least for rendering strings using fonts into windows, and converting
> between code pages correctly.  This was around Windows-2000 or XP,
> certainly well before Windows 7.

I don't need my text editor to absorb all the complexity of a word
processor.


> 
> I think the file system is technically still USC-2 in that it doesn't
> care what the code points mean and doesn't reject improper use of
> surrogate pairs as errors.
> 

[toc] | [prev] | [next] | [standalone]

#6577

On Jun 7, 11:55 am, Jeff Flinn <TriumphSprint2...@hotmail.com> wrote:

> Check the boost mailing list. There's been some discussion recently, as
> well as a presentation at boostcon last month.
>

I found a Boost.Locale library proposed but not in the current release
of Boost.  I did not find any Boostcon slides or videos concerning
it.  Can you be more specific, or give a URL?

Thanks,
—John

[toc] | [prev] | [next] | [standalone]

#6367

On Jun 7, 9:08 am, "John M. Dlugosz" <uwqejpn...@snkmail.com> wrote:
> I may soon be working on a code base that is undergoing
> "internationalization", and what is already done on low levels of
> operating system interaction is using UTF-8 in char-based strings.
>
> Clearly, new code needs to use Unicode, or _some_ way of representing
> a lot more than 256 characters.  UTF-8 and Unicode has merit as an
> encoding and transmission format, and I'm certainly not against it.
>
> But std::string models a sequence of *characters*, and that doesn't
> fit with multi-byte variable-length character encodings of any kind.
> Furthermore, one character won't fit in a char.
>
> Before I get in too deep with my thoughts, I want to see what others
> are doing.  Is there any existing information on this topic?  Is there
> a better place to discuss it?
>
> My initial thoughts is that most of the time the code simply hangs
> onto the string data and doesn't manipulate it, so "funny" contents
> won't matter much.  Furthermore, UTF-8 avoids (by design) many of the
> problems found with multi-byte encodings, so a naive handling might
> work "well enough" for common tasks.  However, I should catalog in
> detail just what works and what the problems are.  For more general
> manipulation of the string, we need functions like the C library's
> mblen etc. and replacements for standard library routines whose
> implementation doesn't handle UTF-8 as multi-byte character strings.
>
> Or, is there some paradigm shift I should be aware of?
>
> —John

John,

I actually had the pleasure(?) of working on some I18n projects in the
past. Most of my work was done in C, so the standard C++ libraries
were not a consideration. However, I think you are correct that a
"naive" implementation will go a long way to accomplishing what you
need. UTF-8 is explicitly designed to be backwards compatible with
ASCII for the first 128 characters.

The challenge, of course, comes when you need to count, manipulate, or
render the text with complex or multibyte characters. I am not aware
of any ubiquitous standard or library that makes this easier. There
are, of course, numerous proprietary libraries available, but I cannot
speak to their quality. Balog probably had the best suggestion to look
into the Glib::ustring library. Frankly, if the UTF-8 structure is
understood, it is fairly trivial to roll your own small library to do
the manipulation. But if, as you said, you are not doing a lot of
manipulation, that consideration is minimal.

All the best,
DiamondBack

[toc] | [prev] | [next] | [standalone]

#6393

On Tue, 2011-06-07, diamondback wrote:
> On Jun 7, 9:08 am, "John M. Dlugosz" <uwqejpn...@snkmail.com> wrote:
>> I may soon be working on a code base that is undergoing
>> "internationalization", and what is already done on low levels of
>> operating system interaction is using UTF-8 in char-based strings.
>>
>> Clearly, new code needs to use Unicode, or _some_ way of representing
>> a lot more than 256 characters.  UTF-8 and Unicode has merit as an
>> encoding and transmission format, and I'm certainly not against it.
...

> The challenge, of course, comes when you need to count, manipulate, or
> render the text with complex or multibyte characters. I am not aware
> of any ubiquitous standard or library that makes this easier. There
> are, of course, numerous proprietary libraries available, but I cannot
> speak to their quality. Balog probably had the best suggestion to look
> into the Glib::ustring library.

'libiconv' is popular and apparently standardized on Unix (i.e. pretty
much everything except Windows).  http://en.wikipedia.org/wiki/Iconv

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .  .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#6377

On Jun 7, 8:08 am, "John M. Dlugosz" <uwqejpn...@snkmail.com> wrote:
> I may soon be working on a code base that is undergoing
> "internationalization", and what is already done on low levels of
> operating system interaction is using UTF-8 in char-based strings.
>
> Clearly, new code needs to use Unicode, or _some_ way of representing
> a lot more than 256 characters.  UTF-8 and Unicode has merit as an
> encoding and transmission format, and I'm certainly not against it.
>
> But std::string models a sequence of *characters*, and that doesn't
> fit with multi-byte variable-length character encodings of any kind.
> Furthermore, one character won't fit in a char.
>
> Before I get in too deep with my thoughts, I want to see what others
> are doing.  Is there any existing information on this topic?  Is there
> a better place to discuss it?
>
> My initial thoughts is that most of the time the code simply hangs
> onto the string data and doesn't manipulate it, so "funny" contents
> won't matter much.  Furthermore, UTF-8 avoids (by design) many of the
> problems found with multi-byte encodings, so a naive handling might
> work "well enough" for common tasks.  However, I should catalog in
> detail just what works and what the problems are.  For more general
> manipulation of the string, we need functions like the C library's
> mblen etc. and replacements for standard library routines whose
> implementation doesn't handle UTF-8 as multi-byte character strings.
>
> Or, is there some paradigm shift I should be aware of?
>
> —John

If you want to work on win32 and unix-like systems, avoid wstring and
wchar like the plague. It's 16 bits on win32, and 32 bits on most unix-
like systems.

String data is itself just data. You can pass it around in std::vector
if you so wanted. The interesting parts come when you need to inspect
and transform the data.

Printing glyphs is well beyond my expertise, and probably beyond
portable C++. That's GUI territory.

Sorting and collation. The same string can be encoded in different
ways, so you need to be able to go to a normalized form to do string
comparison. Also sorting is non-trivial. Sorting is language and
situation dependent, and a lot of the sort orders are /not/
lexicographic. That is, you cannot simply do a memcmp. (At least not
on the strings themselves. There are libraries and such that can puts
strings into another "normalized" form, which differs depending on the
sort order, in which you can do comparisons with memcmp.) For example,
the normal German sort order is slightly different than the German
phonebook sort order. IIRC, in French, you first sort the strings left
to right ignoring accents, and for strings differing in accents only,
you sort /right to left/ based on accents. It's non-trivial if you
want to meet user expectations.

String concatenation and substringing. Encoding units are not Unicode
code points are not "user perceived characters" aka "grapheme
clusters". Latin letter e with accent acute, é, can be encoded in two
different ways, 1 Unicode code point precombined character Latin
letter e with accent acute, and with two Unicode code points: Latin
letter e, and combining character accent acute. Thus, if your program
wants to manipulate data based on user requests, such as "remove the
last character", you need to be careful what "character" means. Does
it mean Unicode code point, or grapheme cluster?

And then there's transformations between encodings. Any good Unicode
library should cover that for you.

That's about all I remember from the top of my head.

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | comp.lang.c++

csiph-web