Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #6354 > unrolled thread

UTF-8 and strings

Back to article view | Back to comp.lang.c++

Contents

  UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-07 08:08 -0700
    Re: UTF-8 and strings Asger-P <junk@asger-p.dk> - 2011-06-07 17:26 +0200
      Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-08 17:58 -0700
        Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-08 18:21 -0700
          Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-09 06:38 -0700
    Re: UTF-8 and strings "Balog Pal" <pasa@lib.hu> - 2011-06-07 17:35 +0200
      Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-08 18:08 -0700
    Re: UTF-8 and strings Jeff Flinn <TriumphSprint2000@hotmail.com> - 2011-06-07 12:55 -0400
      Re: UTF-8 and strings Paavo Helde <myfirstname@osa.pri.ee> - 2011-06-07 15:29 -0500
        Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-08 07:15 -0500
          Re: UTF-8 and strings yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-06-08 15:45 +0000
            Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-08 12:58 -0500
              Re: UTF-8 and strings yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-06-09 09:18 +0000
                Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-09 07:36 -0500
                  Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-10 13:18 -0700
                    Re: UTF-8 and strings Ruben Safir <mrbrklyn@panix.com> - 2011-06-11 00:29 +0000
      Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-10 12:58 -0700
    Re: UTF-8 and strings diamondback <christopher.rau@gmail.com> - 2011-06-07 10:57 -0700
      Re: UTF-8 and strings Jorgen Grahn <grahn+nntp@snipabacken.se> - 2011-06-08 06:38 +0000
    Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-07 13:30 -0700
      Re: UTF-8 and strings Jorgen Grahn <grahn+nntp@snipabacken.se> - 2011-06-08 06:52 +0000
        Re: UTF-8 and strings Marc <marc.glisse@gmail.com> - 2011-06-08 08:38 +0000
          Re: UTF-8 and strings Miles Bader <miles@gnu.org> - 2011-06-08 18:02 +0900
            Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-08 02:05 -0700
              Re: UTF-8 and strings Miles Bader <miles@gnu.org> - 2011-06-08 18:11 +0900
            Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-08 07:22 -0500
          Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-08 02:03 -0700
            Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-08 02:06 -0700
          Re: UTF-8 and strings tm <thomas.mertes@gmx.at> - 2011-06-08 02:59 -0700
          Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-08 18:20 -0700
        Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-08 01:56 -0700
        Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-08 07:19 -0500
          Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-10 13:12 -0700
            Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-10 18:34 -0500
              Re: UTF-8 and strings Miles Bader <miles@gnu.org> - 2011-06-11 13:12 +0900
              Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-12 20:52 -0700
                Re: UTF-8 and strings ruben safir <ruben@mrbrklyn.com> - 2011-06-13 03:57 +0000
                  Re: UTF-8 and strings Asger-P <junk@asger-p.dk> - 2011-06-13 10:51 +0200
      Re: UTF-8 and strings Nobody <nobody@nowhere.com> - 2011-06-08 08:01 +0100
    Re: UTF-8 and strings Nobody <nobody@nowhere.com> - 2011-06-07 21:35 +0100
      Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-08 01:56 -0700
        Re: UTF-8 and strings Nobody <nobody@nowhere.com> - 2011-06-09 03:22 +0100
    Re: UTF-8 and strings yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-06-08 15:55 +0000
      Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-08 17:55 -0700
        Re: UTF-8 and strings yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-06-09 10:02 +0000
          Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-09 08:41 -0500
            Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-12 20:57 -0700
              Re: UTF-8 and strings ruben safir <ruben@mrbrklyn.com> - 2011-06-13 04:46 +0000
                Re: UTF-8 and strings Asger-P <junk@asger-p.dk> - 2011-06-13 11:58 +0200
          Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-09 06:52 -0700

Page 3 of 3 — ← Prev page 1 2 [3]

#6403

On Jun 7, 1:35 pm, Nobody <nob...@nowhere.com> wrote:
> On Tue, 07 Jun 2011 08:08:02 -0700, John M. Dlugosz wrote:
> > But std::string models a sequence of *characters*, and that doesn't
> > fit with multi-byte variable-length character encodings of any kind.
> > Furthermore, one character won't fit in a char.
>
> > Before I get in too deep with my thoughts, I want to see what others
> > are doing.  Is there any existing information on this topic?  Is there
> > a better place to discuss it?
>
> If you want to manipulate "characters", then use std::wstring internally.
> For converting between std::string and std::wstring, possible approaches
> include:
>
> 1. Writing the code yourself (simple if you only use UTF-8)
> 2. mbstowcs and wcstombs (C99, <cstdlib>).
> 3. iconv (Unix)
> 4. MultiByteToWideChar and WideCharToMultiByte (Windows)
>
> Each has at least one drawback. 1 requires that you write the encoding
> and decoding functions. 2 requires setting the locale to a UTF-8 locale
> before any conversion then changing it back afterwards. 3 and 4 aren't
> portable.

I don't see ICU listed, which to my little knowledge is the default
solution for serious Unicode string processing and manipulation.

[toc] | [prev] | [next] | [standalone]

#6470

On Wed, 08 Jun 2011 01:56:43 -0700, Joshua Maurice wrote:

> I don't see ICU listed, which to my little knowledge is the default
> solution for serious Unicode string processing and manipulation.

Too much overhead if you don't need "serious" Unicode support. Which most
applications don't; they just want to support it to the level that the OS
and external libraries do.

[toc] | [prev] | [next] | [standalone]

#6428

In article <c51336b5-cc02-4db9-8f7a-31792552a2da@c41g2000yqm.googlegroups.com>,
John M. Dlugosz <uwqejpnp92@snkmail.com> wrote:
>I may soon be working on a code base that is undergoing
>"internationalization", and what is already done on low levels of
>operating system interaction is using UTF-8 in char-based strings.
>
>Clearly, new code needs to use Unicode, or _some_ way of representing
>a lot more than 256 characters.  UTF-8 and Unicode has merit as an
>encoding and transmission format, and I'm certainly not against it.

You need to get your terminology right and understand clearly the
differences between Unicode, UTF-8, UTF-16, UTF-32 and UCS-2.  You
don't use "Unicode" in an application.  You might use what Windows
refer to as "Unicode" but it will likely be UTF-16 or UCS-2 in
reality.   

>But std::string models a sequence of *characters*, and that doesn't
>fit with multi-byte variable-length character encodings of any kind.
>Furthermore, one character won't fit in a char.
>
>Before I get in too deep with my thoughts, I want to see what others
>are doing.  Is there any existing information on this topic?  Is there
>a better place to discuss it?
>
>My initial thoughts is that most of the time the code simply hangs
>onto the string data and doesn't manipulate it, so "funny" contents
>won't matter much.  Furthermore, UTF-8 avoids (by design) many of the
>problems found with multi-byte encodings, so a naive handling might
>work "well enough" for common tasks.  However, I should catalog in
>detail just what works and what the problems are.  For more general
>manipulation of the string, we need functions like the C library's
>mblen etc. and replacements for standard library routines whose
>implementation doesn't handle UTF-8 as multi-byte character strings.
>
>Or, is there some paradigm shift I should be aware of?


What will you do with you data?
What will you be interfacing with mostly with your internationalised
data? 
What's your OS?

I can't see the point in coverting most of the time.  If your main
purpose is to manipulate, search, transform, reorder strings, then the
convenience of UTF-32 might trump it.  If you are sending data over
the network, you probably should use UTF-8.  If you are interfacing
with an OS often, you probably would be better sticking with the
common Unicode representation used by that OS.  If you are often
interfacing with a GUI library, you may as well keep the data in the
same format.  

[toc] | [prev] | [next] | [standalone]

#6460

On Jun 8, 10:55 am, yatremblay@bel1lin202.(none) (Yannick Tremblay)
wrote:
> In article <c51336b5-cc02-4db9-8f7a-31792552a...@c41g2000yqm.googlegroups.com>,
> John M. Dlugosz <uwqejpn...@snkmail.com> wrote:
>
> >I may soon be working on a code base that is undergoing
> >"internationalization", and what is already done on low levels of
> >operating system interaction is using UTF-8 in char-based strings.
>
> >Clearly, new code needs to use Unicode, or _some_ way of representing
> >a lot more than 256 characters.  UTF-8 and Unicode has merit as an
> >encoding and transmission format, and I'm certainly not against it.
>
> You need to get your terminology right and understand clearly the
> differences between Unicode, UTF-8, UTF-16, UTF-32 and UCS-2.  You
> don't use "Unicode" in an application.  You might use what Windows
> refer to as "Unicode" but it will likely be UTF-16 or UCS-2 in
> reality.  
>
I'm quite aware of the precise terminology.  To summarize, Unicode is
a mapping of cataloged characters to ordinal values, and unlike older
catalogs does not imply any specific means of representing a list of
integers in files or memory.  UTF-8, OTOH, is a specific encoding of
such a list as a sequence of bytes.

What in my message makes it seem that I don't know the difference?  I
don't see any sloppy use of the terminology, and I certainly used my
terms rigorously and precisely in the parts you quoted.

> What will you do with you data?

Anything and everything that applications do with strings.  It's not a
publishing system or word processor, so strings are mainly incidental:
labels, file names, and values used in the GUI.

> What will you be interfacing with mostly with your internationalised
> data?

Just the OS primitives and other like-minded modules.

> What's your OS?

It's portable code that needs to work on a variety of OSs.


> If you are interfacing
> with an OS often, you probably would be better sticking with the
> common Unicode representation used by that OS.  If you are often
> interfacing with a GUI library, you may as well keep the data in the
> same format.  

The problem is that "common code" needs to be applicable to all
operating systems.  Anything facing the OS will be abstracted, using
standard library classes or modules made for the purpose.  The code
base already uses UTF-8 internally and I'm exploring the ramifications
of that, and how to do it correctly.

—John

[toc] | [prev] | [next] | [standalone]

#6478

In article <319effb5-bf19-4c21-86e8-e4f1f4350dfc@p21g2000yqh.googlegroups.com>,
John M. Dlugosz <uwqejpnp92@snkmail.com> wrote:
>On Jun 8, 10:55am, yatremblay@bel1lin202.(none) (Yannick Tremblay)
>wrote:
>> In article <c51336b5-cc02-4db9-8f7a-31792552a...@c41g2000yqm.googlegroups.com>,
>> John M. Dlugosz <uwqejpn...@snkmail.com> wrote:
>>
>> >I may soon be working on a code base that is undergoing
>> >"internationalization", and what is already done on low levels of
>> >operating system interaction is using UTF-8 in char-based strings.
>>
>> >Clearly, new code needs to use Unicode, or _some_ way of representing
>> >a lot more than 256 characters. UTF-8 and Unicode has merit as an
>> >encoding and transmission format, and I'm certainly not against it.
>>
>> You need to get your terminology right and understand clearly the
>> differences between Unicode, UTF-8, UTF-16, UTF-32 and UCS-2. You
>> don't use "Unicode" in an application. You might use what Windows
>> refer to as "Unicode" but it will likely be UTF-16 or UCS-2 in
>> reality. 
>>
>I'm quite aware of the precise terminology.  To summarize, Unicode is
>a mapping of cataloged characters to ordinal values, and unlike older
>catalogs does not imply any specific means of representing a list of
>integers in files or memory.  UTF-8, OTOH, is a specific encoding of
>such a list as a sequence of bytes.
>
>What in my message makes it seem that I don't know the difference?  I
>don't see any sloppy use of the terminology, and I certainly used my
>terms rigorously and precisely in the parts you quoted.

Sorry then.  I read the statements that "already ... is using UTF-8", 
"...new code needs to use Unicode" and  "UTF-8 and Unicode has merit
..." as if it you meant that there were mutually exclusive
alternatives: "Unicode" or "UTF-8".  Apologies

Drifting here but using Unicode is a no-brainer.  Don't even think
about fussing about with Windows codepages, iso-8859-?, shift-jis,
euc-jp, euc-cn, iso-2022-??, etc.  These are enough to give you
nightmares.   So this leaves you only with which representation of
Unicode you should use.  Three posible candidates: UTF-8, UTF-32 and
UTF-16. 

>> What will you do with you data?
>
>Anything and everything that applications do with strings.  It's not a
>publishing system or word processor, so strings are mainly incidental:
>labels, file names, and values used in the GUI.

The the benefits of constant-width characters is not very important
for you.  

>> What will you be interfacing with mostly with your internationalised
>> data?
>
>Just the OS primitives and other like-minded modules.
>
>> What's your OS?
>
>It's portable code that needs to work on a variety of OSs.

OK, so you will have to convert anyway.  Windows API tend to prefer
UTF-16 and Posix tend to lean toward UTF-8 or UTF-32. 

>> If you are interfacing
>> with an OS often, you probably would be better sticking with the
>> common Unicode representation used by that OS. If you are often
>> interfacing with a GUI library, you may as well keep the data in the
>> same format. 
>
>The problem is that "common code" needs to be applicable to all
>operating systems.  Anything facing the OS will be abstracted, using
>standard library classes or modules made for the purpose.  The code
>base already uses UTF-8 internally and I'm exploring the ramifications
>of that, and how to do it correctly.

This sounds like sound design.
I can't see any reason why choosing either of the UTF-* would cause
any problems.  You choose what is your internal representation and you
stay fully consistent with it internally.  Any external input is
converted immediately to the internal representation if needed.  Any
output is converted if needed by the external function wrapper but
your internal API uses your internal representation.  Check iconv for
converting library.

So with this design, there should be no problem of using any of the
UTF-* for internal representation.  What you are left with is what do
you do with the data internally:

You don't really need constant-width character. So one of the main
benefit of UTF-32 is not needed 

You already have some UTF-8 code implemented. So maybe there's no
point changing for changing 

It sounds like you will be converting an existing codebase that was
i18n unaware. Quite often, such codebase is not compatible with
wide-characters and the presence of embedded nulls in a byte stream
could break it.  IMO, it is much easier to gradually switch an
existing codebase from ASCII (or ISO-8859-x) to utf-8 than it is to
change to UTF-16 or UTF-32.  

Does storage space for large amount of string data matter to you?  If
so, do you expect to store a lot of Asian characters?

Regards
Yannick

[toc] | [prev] | [next] | [standalone]

#6494

none (Yannick Tremblay) wrote:

> Drifting here but using Unicode is a no-brainer.  Don't even think
> about fussing about with Windows codepages, iso-8859-?, shift-jis,
> euc-jp, euc-cn, iso-2022-??, etc.  These are enough to give you
> nightmares.   So this leaves you only with which representation of
> Unicode you should use.  Three posible candidates: UTF-8, UTF-32 and
> UTF-16.

But if you KNOW that all you need is what's in the BMP, why not exploit 
that, right?

[toc] | [prev] | [next] | [standalone]

#6668

On Jun 9, 8:41 am, "MikeP" <mp011...@some.org> wrote:

>
> But if you KNOW that all you need is what's in the BMP, why not exploit
> that, right?

Sure, the project is specified to be nationalized into 7 languages,
and they all happen to be serviced by the Latin-1 character set.  So
you decide to use 8-bit chars and assume the Windows program is
running on a system that uses code page 1252 as the default for a
process.

Then one day the boss comes in and says that the next version will be
marketed to China as well.

It is my experience that software projects only get more complex over
time.  Plan for it, unless you are planning to be unsuccessful.

—John

[toc] | [prev] | [next] | [standalone]

#6672

On Sun, 12 Jun 2011 20:57:38 -0700, John M. Dlugosz wrote:

> On Jun 9, 8:41 am, "MikeP" <mp011...@some.org> wrote:
> 
> 
>> But if you KNOW that all you need is what's in the BMP, why not exploit
>> that, right?
> 
> Sure, the project is specified to be nationalized into 7 languages, and
> they all happen to be serviced by the Latin-1 character set.  So you
> decide to use 8-bit chars and assume the Windows program is running on a
> system that uses code page 1252 as the default for a process.
> 
> Then one day the boss comes in and says that the next version will be
> marketed to China as well.
> 
> It is my experience that software projects only get more complex over
> time.  Plan for it, unless you are planning to be unsuccessful.
> 
> —John



Strangely enough, this is a specific problem for a specific kind of app, 
like a word processor.  IT people even in china learn rudimentary English 
because scanf doesn't translate to Chinese wide chars.  If comp sci 
originally were devised and discovered in china, it would have been a 
bummer for an 8 bit processor with limited memory, and a limit of 256 
chars.

 

[toc] | [prev] | [next] | [standalone]

#6686

Hi ruben

On the: 13. of juni-2011 At: 06:46 ruben safir wrote:

> On Sun, 12 Jun 2011 20:57:38 -0700, John M. Dlugosz wrote:
>
>> On Jun 9, 8:41 am, "MikeP" <mp011...@some.org> wrote:
>>
>>
>>> But if you KNOW that all you need is what's in the BMP, why not exploit
>>> that, right?
>>
>> Sure, the project is specified to be nationalized into 7 languages, and
>> they all happen to be serviced by the Latin-1 character set.  So you
>> decide to use 8-bit chars and assume the Windows program is running on a
>> system that uses code page 1252 as the default for a process.
>>
>> Then one day the boss comes in and says that the next version will be
>> marketed to China as well.
>>
>> It is my experience that software projects only get more complex over
>> time.  Plan for it, unless you are planning to be unsuccessful.
>>
>> —John
>
>
>
> Strangely enough, this is a specific problem for a specific kind of app,
> like a word processor.

I think You are narrowing it a bit to much.
Most applications that interact with the user and their keyboard
need to consider codepages at some level, if they want to be used
outside the region where they were designed.

A simple thing like comparing two strings case insensitive will often
not work on non ASCII characters if You use the standard c functions.

This is an interesting page to read:
http://cppcms.sourceforge.net/boost_locale/html/appendix.html

If You live in a country where english is the language You
probably haven't seen the errors Your self, but fortunately
I live in Denmark so I have had to deal with this issue from
day one.


Best regards
Asger-P

[toc] | [prev] | [next] | [standalone]

#6499

On Jun 9, 5:02 am, yatremblay@bel1lin202.(none) (Yannick Tremblay)
wrote:
> Sorry then.  I read the statements that "already ... is using UTF-8",
> "...new code needs to use Unicode" and  "UTF-8 and Unicode has merit
> ..." as if it you meant that there were mutually exclusive
> alternatives: "Unicode" or "UTF-8".  Apologies
>

I see.  I meant them separately, "Unicode has merit" and "UTF-8 has
merit", not as synonyms, and not as mutual exclusive either.  Using
Unicode (as opposed to GB18030, local code pages, ISO-2022 code page
switching) is good, and using UTF-8 (as opposed to UTF-16 etc.) for
internal representation and API passing has its points.  In Windows,
I've moved towards using UTF-16 and wide strings.  This project is
different.

—John

[toc] | [prev] | [standalone]

Page 3 of 3 — ← Prev page 1 2 [3]

Back to top | Article view | comp.lang.c++

csiph-web