Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #6354 > unrolled thread

UTF-8 and strings

Back to article view | Back to comp.lang.c++

Contents

  UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-07 08:08 -0700
    Re: UTF-8 and strings Asger-P <junk@asger-p.dk> - 2011-06-07 17:26 +0200
      Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-08 17:58 -0700
        Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-08 18:21 -0700
          Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-09 06:38 -0700
    Re: UTF-8 and strings "Balog Pal" <pasa@lib.hu> - 2011-06-07 17:35 +0200
      Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-08 18:08 -0700
    Re: UTF-8 and strings Jeff Flinn <TriumphSprint2000@hotmail.com> - 2011-06-07 12:55 -0400
      Re: UTF-8 and strings Paavo Helde <myfirstname@osa.pri.ee> - 2011-06-07 15:29 -0500
        Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-08 07:15 -0500
          Re: UTF-8 and strings yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-06-08 15:45 +0000
            Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-08 12:58 -0500
              Re: UTF-8 and strings yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-06-09 09:18 +0000
                Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-09 07:36 -0500
                  Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-10 13:18 -0700
                    Re: UTF-8 and strings Ruben Safir <mrbrklyn@panix.com> - 2011-06-11 00:29 +0000
      Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-10 12:58 -0700
    Re: UTF-8 and strings diamondback <christopher.rau@gmail.com> - 2011-06-07 10:57 -0700
      Re: UTF-8 and strings Jorgen Grahn <grahn+nntp@snipabacken.se> - 2011-06-08 06:38 +0000
    Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-07 13:30 -0700
      Re: UTF-8 and strings Jorgen Grahn <grahn+nntp@snipabacken.se> - 2011-06-08 06:52 +0000
        Re: UTF-8 and strings Marc <marc.glisse@gmail.com> - 2011-06-08 08:38 +0000
          Re: UTF-8 and strings Miles Bader <miles@gnu.org> - 2011-06-08 18:02 +0900
            Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-08 02:05 -0700
              Re: UTF-8 and strings Miles Bader <miles@gnu.org> - 2011-06-08 18:11 +0900
            Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-08 07:22 -0500
          Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-08 02:03 -0700
            Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-08 02:06 -0700
          Re: UTF-8 and strings tm <thomas.mertes@gmx.at> - 2011-06-08 02:59 -0700
          Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-08 18:20 -0700
        Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-08 01:56 -0700
        Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-08 07:19 -0500
          Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-10 13:12 -0700
            Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-10 18:34 -0500
              Re: UTF-8 and strings Miles Bader <miles@gnu.org> - 2011-06-11 13:12 +0900
              Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-12 20:52 -0700
                Re: UTF-8 and strings ruben safir <ruben@mrbrklyn.com> - 2011-06-13 03:57 +0000
                  Re: UTF-8 and strings Asger-P <junk@asger-p.dk> - 2011-06-13 10:51 +0200
      Re: UTF-8 and strings Nobody <nobody@nowhere.com> - 2011-06-08 08:01 +0100
    Re: UTF-8 and strings Nobody <nobody@nowhere.com> - 2011-06-07 21:35 +0100
      Re: UTF-8 and strings Joshua Maurice <joshuamaurice@gmail.com> - 2011-06-08 01:56 -0700
        Re: UTF-8 and strings Nobody <nobody@nowhere.com> - 2011-06-09 03:22 +0100
    Re: UTF-8 and strings yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-06-08 15:55 +0000
      Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-08 17:55 -0700
        Re: UTF-8 and strings yatremblay@bel1lin202.(none) (Yannick Tremblay) - 2011-06-09 10:02 +0000
          Re: UTF-8 and strings "MikeP" <mp011011@some.org> - 2011-06-09 08:41 -0500
            Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-12 20:57 -0700
              Re: UTF-8 and strings ruben safir <ruben@mrbrklyn.com> - 2011-06-13 04:46 +0000
                Re: UTF-8 and strings Asger-P <junk@asger-p.dk> - 2011-06-13 11:58 +0200
          Re: UTF-8 and strings "John M. Dlugosz" <uwqejpnp92@snkmail.com> - 2011-06-09 06:52 -0700

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#6395

On Tue, 2011-06-07, Joshua Maurice wrote:
> On Jun 7, 8:08 am, "John M. Dlugosz" <uwqejpn...@snkmail.com> wrote:
>> I may soon be working on a code base that is undergoing
>> "internationalization", and what is already done on low levels of
>> operating system interaction is using UTF-8 in char-based strings.
>>
>> Clearly, new code needs to use Unicode, or _some_ way of representing
>> a lot more than 256 characters.  UTF-8 and Unicode has merit as an
>> encoding and transmission format, and I'm certainly not against it.
>>
>> But std::string models a sequence of *characters*, and that doesn't
>> fit with multi-byte variable-length character encodings of any kind.
>> Furthermore, one character won't fit in a char.
>>
>> Before I get in too deep with my thoughts, I want to see what others
>> are doing.  Is there any existing information on this topic?  Is there
>> a better place to discuss it?
>>
>> My initial thoughts is that most of the time the code simply hangs
>> onto the string data and doesn't manipulate it, so "funny" contents
>> won't matter much.  Furthermore, UTF-8 avoids (by design) many of the
>> problems found with multi-byte encodings, so a naive handling might
>> work "well enough" for common tasks.  However, I should catalog in
>> detail just what works and what the problems are.  For more general
>> manipulation of the string, we need functions like the C library's
>> mblen etc. and replacements for standard library routines whose
>> implementation doesn't handle UTF-8 as multi-byte character strings.
>>
>> Or, is there some paradigm shift I should be aware of?
>>
>> ?John
>
> If you want to work on win32 and unix-like systems, avoid wstring and
> wchar like the plague. It's 16 bits on win32, and 32 bits on most unix-
> like systems.

If you mean the 16/32 difference is the reason to avoid wchar, why
exactly?  I don't always need to know the exact size of a long; why
should wchar be any different?

Of course, if the trend is "let the normal representation of strings
be UTF-8" then the answer doesn't matter.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .  .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#6401

Jorgen Grahn  wrote:

> On Tue, 2011-06-07, Joshua Maurice wrote:
>> If you want to work on win32 and unix-like systems, avoid wstring and
>> wchar like the plague. It's 16 bits on win32, and 32 bits on most unix-
>> like systems.
> 
> If you mean the 16/32 difference is the reason to avoid wchar, why
> exactly?  I don't always need to know the exact size of a long; why
> should wchar be any different?

32 bits are enough that any unicode character fits in a single
wchat_t, so you can work on those almost (ok, that's a big "almost")
as easily as with plain old ascii. 16 bits force you to use some
variable length encoding like utf-16, so this is just as complicated
as utf-8.

[toc] | [prev] | [next] | [standalone]

#6404

Marc <marc.glisse@gmail.com> writes:
>> If you mean the 16/32 difference is the reason to avoid wchar, why
>> exactly?  I don't always need to know the exact size of a long; why
>> should wchar be any different?
>
> 32 bits are enough that any unicode character fits in a single
> wchat_t, so you can work on those almost (ok, that's a big "almost")
> as easily as with plain old ascii. 16 bits force you to use some
> variable length encoding like utf-16, so this is just as complicated
> as utf-8.

Exactly.  utf-16 offers no simplicity advantage over utf-8, and suffers
from some significant disadvantages.

In practice, I suppose that many windows apps probably just ignore
anything outside the BMP, pretend "it's all 16-bit!", and as a result
suffer from mysterious and bizarre bugs when such characters crop up...

-Miles

-- 
Accord, n. Harmony.

[toc] | [prev] | [next] | [standalone]

#6406

On Jun 8, 2:02 am, Miles Bader <mi...@gnu.org> wrote:
> Marc <marc.gli...@gmail.com> writes:
> >> If you mean the 16/32 difference is the reason to avoid wchar, why
> >> exactly?  I don't always need to know the exact size of a long; why
> >> should wchar be any different?
>
> > 32 bits are enough that any unicode character fits in a single
> > wchat_t, so you can work on those almost (ok, that's a big "almost")
> > as easily as with plain old ascii. 16 bits force you to use some
> > variable length encoding like utf-16, so this is just as complicated
> > as utf-8.
>
> Exactly.  utf-16 offers no simplicity advantage over utf-8, and suffers
> from some significant disadvantages.

I wouldn't say that. That's very a very Europe-centric view. UTF-16
results in smaller memory footprints for a variety of Asian scripts
compared to UTF-8.

[toc] | [prev] | [next] | [standalone]

#6409

Joshua Maurice <joshuamaurice@gmail.com> writes:
>> Exactly.  utf-16 offers no simplicity advantage over utf-8, and suffers
>> from some significant disadvantages.
>
> I wouldn't say that. That's very a very Europe-centric view. UTF-16
> results in smaller memory footprints for a variety of Asian scripts
> compared to UTF-8.

That's why I said "no simplicity advantage" instead of "no
advantage"... :]

In practice I don't think the space savings are worth the pain for most
apps, even if dealing with lots of CJK or something.

-Miles

-- 
Liberty, n. One of imagination's most precious possessions.

[toc] | [prev] | [next] | [standalone]

#6418

Miles Bader wrote:

> Exactly.  utf-16 offers no simplicity advantage over utf-8, and
> suffers from some significant disadvantages.

"Given" (is it a given?) that "UTF-16" is likely to in reality be UCS-2 
(?), then much simpler. And what are the consequences of making that 
assumption?

>
> In practice, I suppose that many windows apps probably just ignore
> anything outside the BMP, pretend "it's all 16-bit!",

Yes, yes!

> and as a result
> suffer from mysterious and bizarre bugs when such characters crop
> up...

Do tell, please.

[toc] | [prev] | [next] | [standalone]

#6405

On Jun 8, 1:38 am, Marc <marc.gli...@gmail.com> wrote:
> Jorgen Grahn  wrote:
> > On Tue, 2011-06-07, Joshua Maurice wrote:
> >> If you want to work on win32 and unix-like systems, avoid wstring and
> >> wchar like the plague. It's 16 bits on win32, and 32 bits on most unix-
> >> like systems.
>
> > If you mean the 16/32 difference is the reason to avoid wchar, why
> > exactly?  I don't always need to know the exact size of a long; why
> > should wchar be any different?
>
> 32 bits are enough that any unicode character fits in a single
> wchat_t, so you can work on those almost (ok, that's a big "almost")
> as easily as with plain old ascii. 16 bits force you to use some
> variable length encoding like utf-16, so this is just as complicated
> as utf-8.

You can work with it almost like ASCII, except for combining character
and multi-code point grapheme clusters.

[toc] | [prev] | [next] | [standalone]

#6407

On Jun 8, 2:03 am, Joshua Maurice <joshuamaur...@gmail.com> wrote:
> On Jun 8, 1:38 am, Marc <marc.gli...@gmail.com> wrote:
>
> > Jorgen Grahn  wrote:
> > > On Tue, 2011-06-07, Joshua Maurice wrote:
> > >> If you want to work on win32 and unix-like systems, avoid wstring and
> > >> wchar like the plague. It's 16 bits on win32, and 32 bits on most unix-
> > >> like systems.
>
> > > If you mean the 16/32 difference is the reason to avoid wchar, why
> > > exactly?  I don't always need to know the exact size of a long; why
> > > should wchar be any different?
>
> > 32 bits are enough that any unicode character fits in a single
> > wchat_t, so you can work on those almost (ok, that's a big "almost")
> > as easily as with plain old ascii. 16 bits force you to use some
> > variable length encoding like utf-16, so this is just as complicated
> > as utf-8.
>
> You can work with it almost like ASCII, except for combining character
> and multi-code point grapheme clusters.

(Hit send too soon.) And of course sorting, equivalence comparisons,
and... nevermind.

[toc] | [prev] | [next] | [standalone]

#6410

On 8 Jun., 10:38, Marc <marc.gli...@gmail.com> wrote:
> Jorgen Grahn  wrote:
> > On Tue, 2011-06-07, Joshua Maurice wrote:
> >> If you want to work on win32 and unix-like systems, avoid wstring and
> >> wchar like the plague. It's 16 bits on win32, and 32 bits on most unix-
> >> like systems.
>
> > If you mean the 16/32 difference is the reason to avoid wchar, why
> > exactly?  I don't always need to know the exact size of a long; why
> > should wchar be any different?
>
> 32 bits are enough that any unicode character fits in a single
> wchat_t, so you can work on those almost (ok, that's a big "almost")
> as easily as with plain old ascii. 16 bits force you to use some
> variable length encoding like utf-16, so this is just as complicated
> as utf-8.

That's the reason I used UTF-32 as encoding in my
libraries. As you said 32 bits are enough that any unicode
character fits. Libraries with UTF-8 encoding need to
distinguish between byte- and character-index. To get
character number 12345 from an UTF-8 string, the string
must be processed from the beginning. Ok, UTF-32 wastes
some space, but I think that many 64-bit integers also
contain values that would fit into a byte. I think that
many programs deal only with strings that have several
thousands up to one million of characters. This is no
problem for todays gigabyte machines.

The hard part of UTF-32 strings is that they need to
be converted every time you talk with the OS. This
needs to be encapsulated since the operating systems
have different ideas what their native string type
should be. For my libraries this work is already done.
Note that the conversions to and from operating system
strings turned out to be NO performace problem.

BTW. The description of my libraries is online. See:

  http://seed7.sourceforge.net/libraries

Sorry to say, but this are not C++ libraries, but the
basic functions are written in C and are licensed with
the GPL. So they might be of use for C++ also.


Greetings Thomas Mertes

--
Seed7 Homepage:  http://seed7.sourceforge.net
Seed7 - The extensible programming language: User defined statements
and operators, abstract data types, templates without special
syntax, OO with interfaces and multiple dispatch, statically typed,
interpreted or compiled, portable, runs under linux/unix/windows.

[toc] | [prev] | [next] | [standalone]

#6465

On Jun 8, 3:38 am, Marc <marc.gli...@gmail.com> wrote:
> Jorgen Grahn  wrote:
> 32 bits are enough that any unicode character fits in a single
> wchat_t, so you can work on those almost (ok, that's a big "almost")
> as easily as with plain old ascii. 16 bits force you to use some
> variable length encoding like utf-16, so this is just as complicated
> as utf-8.

That's my feeling exactly.  UTF-16 is "neither here nor there" in that
you still can't assume one cell per code point but have to deal with
surrogate pairs, and it takes up more memory for ASCII and Western
languages, and opens up byte-ordering issues.

OTOH, manipulating "characters" doesn't necessarily mean 1:1 with
"code points" anyway, as someone else pointed out.  I'm not worried
about that now, since the project isn't a word processor and any use
of Backspace will be in the platform-specific GUI components.

—John

[toc] | [prev] | [next] | [standalone]

#6402

On Jun 7, 11:52 pm, Jorgen Grahn <grahn+n...@snipabacken.se> wrote:
> On Tue, 2011-06-07, Joshua Maurice wrote:
> > On Jun 7, 8:08 am, "John M. Dlugosz" <uwqejpn...@snkmail.com> wrote:
> >> I may soon be working on a code base that is undergoing
> >> "internationalization", and what is already done on low levels of
> >> operating system interaction is using UTF-8 in char-based strings.
>
> >> Clearly, new code needs to use Unicode, or _some_ way of representing
> >> a lot more than 256 characters. UTF-8 and Unicode has merit as an
> >> encoding and transmission format, and I'm certainly not against it.
>
> >> But std::string models a sequence of *characters*, and that doesn't
> >> fit with multi-byte variable-length character encodings of any kind.
> >> Furthermore, one character won't fit in a char.
>
> >> Before I get in too deep with my thoughts, I want to see what others
> >> are doing. Is there any existing information on this topic? Is there
> >> a better place to discuss it?
>
> >> My initial thoughts is that most of the time the code simply hangs
> >> onto the string data and doesn't manipulate it, so "funny" contents
> >> won't matter much. Furthermore, UTF-8 avoids (by design) many of the
> >> problems found with multi-byte encodings, so a naive handling might
> >> work "well enough" for common tasks. However, I should catalog in
> >> detail just what works and what the problems are. For more general
> >> manipulation of the string, we need functions like the C library's
> >> mblen etc. and replacements for standard library routines whose
> >> implementation doesn't handle UTF-8 as multi-byte character strings.
>
> >> Or, is there some paradigm shift I should be aware of?
>
> >> ?John
>
> > If you want to work on win32 and unix-like systems, avoid wstring and
> > wchar like the plague. It's 16 bits on win32, and 32 bits on most unix-
> > like systems.
>
> If you mean the 16/32 difference is the reason to avoid wchar, why
> exactly?  I don't always need to know the exact size of a long; why
> should wchar be any different?
>
> Of course, if the trend is "let the normal representation of strings
> be UTF-8" then the answer doesn't matter.

If the goal is UTF-8 encoding, then obviously wstring is a bad choice.
If the goal is constant width encoding, then that's UTF-32, which is
not wstring. You ought to use std::basic_string specialized on
uint32_t or whatever the name is for a guaranteed 32 bit character
type in C++0x. (I'd rather roll my own honestly, but meh.)

If you're not going to inspect or manipulate the string data, might as
well use vector<char>. If you are going to do string-y things, you
probably want to rely on a particular encoding, though I suppose it's
possible to do the manipulations agnostic to whether it's UTF-8 or
UTF-16. If it's UTF-32, I suspect there'd be too much reliance on the
assumption that it's a constant width encoding of Unicode code
points.

Really though, the main reason I said that is I don't see the use case
for having UTF-16 on some platforms, and UTF-32 on other platforms.
There's no requirement to do that. It makes no sense to do that. This
is different than how big is long. You're not using arrays of longs to
stored variable width encodings (or constant width for  UTF-32) of
data. Here you are. The exact size does kind of matter.

[toc] | [prev] | [next] | [standalone]

#6417

Jorgen Grahn wrote:

> Of course, if the trend is "let the normal representation of strings
> be UTF-8" then the answer doesn't matter.
>

What is this/the enamourment with UTF-8 anyway? 

[toc] | [prev] | [next] | [standalone]

#6581

On Jun 8, 7:19 am, "MikeP" <mp011...@some.org> wrote:
> Jorgen Grahn wrote:
> > Of course, if the trend is "let the normal representation of strings
> > be UTF-8" then the answer doesn't matter.
>
> What is this/the enamourment with UTF-8 anyway?

There are a few:

(1)  Old code is written using byte-size character strings of *some*
kind.  For languages larger character sets, the existing practice is
to use a "multi-byte" code page, and actually have variable length
where ASCII is still a single byte and certain ranges of character
indicate prefixes or paired use.

So lots of code uses byte strings with "internationalization" meaning
to allow the target system to specify the mapping of which character
is what value, and allowing multi-byte characters mixed with single
byte characters.

So, using UTF-8 fits the existing data types and code.  It's just
another "code page" to such code.

(2)  It is efficient for non-ideographic languages, and data-
processing data that uses mostly ASCII identifiers.

(3)  Being defined as a series of bytes, it is byte-order neutral.

(4)  Contrast with UTF-16, which _still_ requires awareness of pairs
to allow more than 64K code values.

So, it's attractive for data storage and transmission.

[toc] | [prev] | [next] | [standalone]

#6596

John M. Dlugosz wrote:
> On Jun 8, 7:19 am, "MikeP" <mp011...@some.org> wrote:
>> Jorgen Grahn wrote:
>>> Of course, if the trend is "let the normal representation of strings
>>> be UTF-8" then the answer doesn't matter.
>>
>> What is this/the enamourment with UTF-8 anyway?
>
> There are a few:
>
> (1)  Old code is written using byte-size character strings of *some*
> kind.  For languages larger character sets, the existing practice is
> to use a "multi-byte" code page, and actually have variable length
> where ASCII is still a single byte and certain ranges of character
> indicate prefixes or paired use.
>
> So lots of code uses byte strings with "internationalization" meaning
> to allow the target system to specify the mapping of which character
> is what value, and allowing multi-byte characters mixed with single
> byte characters.
>
> So, using UTF-8 fits the existing data types and code.  It's just
> another "code page" to such code.
>
> (2)  It is efficient for non-ideographic languages, and data-
> processing data that uses mostly ASCII identifiers.
>
> (3)  Being defined as a series of bytes, it is byte-order neutral.
>
> (4)  Contrast with UTF-16, which _still_ requires awareness of pairs
> to allow more than 64K code values.
>
> So, it's attractive for data storage and transmission.

I guess I have a hard time seeing how anything multi-byte is a boon. But, 
and it's a big but (not to be confused with a phat azz!), if one doesn't 
need "internationalization" (I mean other than English), it's a waste of 
effort. Yes? 

[toc] | [prev] | [next] | [standalone]

#6604

"MikeP" <mp011011@some.org> writes:
> I guess I have a hard time seeing how anything multi-byte is a boon. But, 
> and it's a big but (not to be confused with a phat azz!), if one doesn't 
> need "internationalization" (I mean other than English), it's a waste of 
> effort. Yes? 

But that's the thing:  if you're just doing things casually, but,
e.g., want to use a few special chars here and there, or allow users
more freedom in what filenames they're allowed to use, then UTF-8
_doesn't_ require much effort, it's a fairly easy tweak to ASCII-only
code.  If the bulk of strings are English, then UTF-8 is also very
space-efficient.

It's UTF-16, which requires even the most trivial parts of
string-handling paths to be completely replaced, that's a pain in the
butt -- and then really offers almost no advantage to offset the
various disadvantages!

The only reasons I can see to use UTF-16, are: (1) you're writing
windows-only code, never expect to port it, and want to fit better
with windows library functions that expect UTF-16 strings, or (2)
you're writing an app to handle absolutely _massive_ amounts of CJK
text, and the space savings for CJK text in UTF-16 compared to UTF-8
are critical for you.

Very, very, few people are doing (2), so basically that leaves (1).

-Miles

-- 
Congratulation, n. The civility of envy.

[toc] | [prev] | [next] | [standalone]

#6666

On Jun 10, 6:34 pm, "MikeP" <mp011...@some.org> wrote:
>
> I guess I have a hard time seeing how anything multi-byte is a boon. But,
> and it's a big but (not to be confused with a phat azz!), if one doesn't
> need "internationalization" (I mean other than English), it's a waste of
> effort. Yes?

Since ASCII is a proper subset of UTF-8, you can write plain English
and get one byte per character.  So there is no special effort on your
part.

It's rare that you would not care about internationalization.  Even if
you don't plan to change your displayed UI into other languages,
people will try using file names and enter strings in their own
language.

[toc] | [prev] | [next] | [standalone]

#6667

On Sun, 12 Jun 2011 20:52:01 -0700, John M. Dlugosz wrote:


> It's rare that you would not care about internationalization.

which is why it took 40 plus years to even think about it...

BTW - what you wrote is actually incorrect.  I'm not an expert on utf-8 
but god knows I've followed enough arguing about it, especially on Rik 
Moens conspire mailing list, to understand this basic fact.

i refer you to Rick for the details..

[toc] | [prev] | [next] | [standalone]

#6685

Hi ruben

On the: 13. of juni-2011 At: 05:57 ruben safir wrote:

> On Sun, 12 Jun 2011 20:52:01 -0700, John M. Dlugosz wrote:
>
>
>> It's rare that you would not care about internationalization.
>
> which is why it took 40 plus years to even think about it...
>
> BTW - what you wrote is actually incorrect.  I'm not an expert on utf-8
> but god knows I've followed enough arguing about it, especially on Rik
> Moens conspire mailing list, to understand this basic fact.

What part is incorrect ?

Have a look at:
http://en.wikipedia.org/wiki/UTF-8
and You will see that the first 127 characters are actually ASCII


Best regards
Asger-P

[toc] | [prev] | [next] | [standalone]

#6396

On Tue, 07 Jun 2011 13:30:30 -0700, Joshua Maurice wrote:

> If you want to work on win32 and unix-like systems, avoid wstring and
> wchar like the plague. It's 16 bits on win32, and 32 bits on most unix-
> like systems.

That's a silly reason to avoid std::wstring and wchar_t. You may as well
suggest avoiding "int" or "long" for similar reasons.

If you need to deal with individual "characters", wchar_t and std::wstring
are the most appropriate solution. Multi-byte strings are only sensible if
you're mainly treating strings as opaque values, passing them around
without paying too much attention to what is inside them.

Most of the awkward issues which affect internationalisation (e.g.
pre-composed characters versus composing characters, lexicographic
ordering, equivalence, etc) apply equally to multi-byte or wide strings,
so avoiding wide strings doesn't gain you anything here.

[toc] | [prev] | [next] | [standalone]

#6378

On Tue, 07 Jun 2011 08:08:02 -0700, John M. Dlugosz wrote:

> But std::string models a sequence of *characters*, and that doesn't
> fit with multi-byte variable-length character encodings of any kind.
> Furthermore, one character won't fit in a char.
> 
> Before I get in too deep with my thoughts, I want to see what others
> are doing.  Is there any existing information on this topic?  Is there
> a better place to discuss it?

If you want to manipulate "characters", then use std::wstring internally.
For converting between std::string and std::wstring, possible approaches
include:

1. Writing the code yourself (simple if you only use UTF-8)
2. mbstowcs and wcstombs (C99, <cstdlib>).
3. iconv (Unix)
4. MultiByteToWideChar and WideCharToMultiByte (Windows)

Each has at least one drawback. 1 requires that you write the encoding
and decoding functions. 2 requires setting the locale to a UTF-8 locale
before any conversion then changing it back afterwards. 3 and 4 aren't
portable.

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | comp.lang.c++

csiph-web