Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #9075 > unrolled thread

Converting from std::wstring to UTF-8 std::string

Back to article view | Back to comp.lang.c++

Contents

  Converting from std::wstring to UTF-8 std::string Peter Poulsen <pgpoulsen@gmail.com> - 2011-08-19 09:34 -0700
    Re: Converting from std::wstring to UTF-8 std::string Victor Bazarov <v.bazarov@comcast.invalid> - 2011-08-19 12:44 -0400
    Re: Converting from std::wstring to UTF-8 std::string Sam <sam@email-scan.com> - 2011-08-19 11:59 -0500
      Re: Converting from std::wstring to UTF-8 std::string Nobody <nobody@nowhere.com> - 2011-08-19 18:21 +0100
    Re: Converting from std::wstring to UTF-8 std::string Paavo Helde <myfirstname@osa.pri.ee> - 2011-08-19 12:48 -0500
      Re: Converting from std::wstring to UTF-8 std::string Joshua Maurice <joshuamaurice@gmail.com> - 2011-08-19 14:13 -0700
        Re: Converting from std::wstring to UTF-8 std::string Marc <marc.glisse@gmail.com> - 2011-08-19 22:09 +0000
    Re: Converting from std::wstring to UTF-8 std::string Nobody <nobody@nowhere.com> - 2011-08-19 19:01 +0100
    Re: Converting from std::wstring to UTF-8 std::string john <gmlrTOGLI@DELTHISlibero.it.DELCAPS> - 2011-08-19 20:37 +0200

#9075 — Converting from std::wstring to UTF-8 std::string

Hi

I'm trying to convert a std::wstring to std::string UTF-8 encoded. I
have made a simple function that does the trick for three letters but
it can hardly be considered a generic solution.

<code>
  std::string wchar2utf8(std::wstring const& wstr)
  {
	std::string str;
	std::wstring::const_iterator itr = wstr.begin();
	while(itr != wstr.end()) {
	  switch(*itr) {
	  case 0x00f8: // ø
		str.push_back(0xc3);
		str.push_back(0xb8);
		break;
	  case 0x00e5: // å
		str.push_back(0xc3);
		str.push_back(0xa5);
		break;
	  case 0x00e6: // æ
		str.push_back(0xc3);
		str.push_back(0xa6);
		break;
	  default:
		str.push_back(*itr);
		break;
	  }
	  ++itr;
	}

	return str;
  }
</code>

I have tried google but it is extremely hard to find a good
explanation of how to do it. Is there somebody that can give an
example of a generic solution for my wchar2utf8() function?

Yours
/peter

[toc] | [next] | [standalone]

#9076

On 8/19/2011 12:34 PM, Peter Poulsen wrote:
> I'm trying to convert a std::wstring to std::string UTF-8 encoded. I
> have made a simple function that does the trick for three letters but
>[...]

Isn't there a library that you could use?  I can't imagine nobody before 
you have thought of the need to do that...

V
-- 
I do not respond to top-posted replies, please don't ask

[toc] | [prev] | [next] | [standalone]

#9077

[Multipart message — attachments visible in raw view] — view raw

Peter Poulsen writes:

> Hi
>
> I'm trying to convert a std::wstring to std::string UTF-8 encoded. I
> have made a simple function that does the trick for three letters but
> it can hardly be considered a generic solution.
>
> <code>
>   std::string wchar2utf8(std::wstring const& wstr)
>   {
> 	std::string str;
> 	std::wstring::const_iterator itr = wstr.begin();
> 	while(itr != wstr.end()) {
> 	  switch(*itr) {
> 	  case 0x00f8: // ø
> 		str.push_back(0xc3);
> 		str.push_back(0xb8);
> 		break;
> 	  case 0x00e5: // å
> 		str.push_back(0xc3);
> 		str.push_back(0xa5);
> 		break;
> 	  case 0x00e6: // æ
> 		str.push_back(0xc3);
> 		str.push_back(0xa6);
> 		break;
> 	  default:
> 		str.push_back(*itr);
> 		break;
> 	  }
> 	  ++itr;
> 	}
>
> 	return str;
>   }
> </code>
>
> I have tried google but it is extremely hard to find a good
> explanation of how to do it. Is there somebody that can give an
> example of a generic solution for my wchar2utf8() function?

You are assuming that wchar_t stores unicode values. They may be, on your  
platform, but this may not necessarily be the case somewhere.

For a portable solution, just instantiate a UTF-8 locale, and use the  
std::codecvt facet to convert wchar_ts to chars. Something like:

std::locale l("en_US.utf-8");

const std::codecvt<wchar_t,char,std::mbstate_t>& myfacet =
    std::use_facet<std::codecvt<wchar_t,char,std::mbstate_t> >(l);

Then use out() to convert your wchar_ts to chars.

[toc] | [prev] | [next] | [standalone]

#9078

On Fri, 19 Aug 2011 11:59:47 -0500, Sam wrote:

> For a portable solution, just instantiate a UTF-8 locale,

Which one?

AFAIK, there is no guarantee that any given platform even has a UTF-8
locale.

[toc] | [prev] | [next] | [standalone]

#9079

Peter Poulsen <pgpoulsen@gmail.com> wrote in news:7ebdb5cb-3555-4657-
9b74-c5af9157af1b@r12g2000vbe.googlegroups.com:

> Hi
> 
> I'm trying to convert a std::wstring to std::string UTF-8 encoded. I
> have made a simple function that does the trick for three letters but
> it can hardly be considered a generic solution.

First you have to know/decide what is stored in std::wstring. Is it UTF-
16 or UCS-2? UTF-16 needs to be first decoded.

It would be a nice exercise to code a pair of functions for UTF-8 
encoding-decoding. It involves some bit manipulation, but nothing too 
hard. Just follow the Wikipedia page for UTF-8. 

Otherwise you can always use some library, iconv is kind of standard on 
Linux and on Windows this functionality is built-in (MultiByteToWideChar
(), WideCharToMultiByte()). 

[naive implementation example snipped]

> 
> I have tried google but it is extremely hard to find a good
> explanation of how to do it. Is there somebody that can give an
> example of a generic solution for my wchar2utf8() function?

I used to have a handwritten routine in the past, if I happen to find it 
I can post it.

Cheers
Paavo

[toc] | [prev] | [next] | [standalone]

#9094

On Aug 19, 10:48 am, Paavo Helde <myfirstn...@osa.pri.ee> wrote:
> Peter Poulsen <pgpoul...@gmail.com> wrote in news:7ebdb5cb-3555-4657-
> 9b74-c5af9157a...@r12g2000vbe.googlegroups.com:
>
> > Hi
>
> > I'm trying to convert a std::wstring to std::string UTF-8 encoded. I
> > have made a simple function that does the trick for three letters but
> > it can hardly be considered a generic solution.
>
> First you have to know/decide what is stored in std::wstring. Is it UTF-
> 16 or UCS-2? UTF-16 needs to be first decoded.

Note that wchar_t is 32 bits on most unix-like operating systems, and
by convention wstrings store UTF-32 strings. Windows is the oddball
desktop/server with 16 bit wchar_t.

[toc] | [prev] | [next] | [standalone]

#9105

Joshua Maurice  wrote:

> Note that wchar_t is 32 bits on most unix-like operating systems, and
> by convention wstrings store UTF-32 strings. Windows is the oddball
> desktop/server with 16 bit wchar_t.

Yes, good thing we have char32_t now (er, soon at least).

[toc] | [prev] | [next] | [standalone]

#9080

On Fri, 19 Aug 2011 09:34:36 -0700, Peter Poulsen wrote:

> Is there somebody that can give an example of a generic solution for my
> wchar2utf8() function?

The following function converts a single Unicode character to UTF-8. Note
that wchar_t may or may not be a Unicode character (particularly on
Windows, which can't make up its mind whether wide strings are UCS-2 or
UTF-16).

#include <string>

void unicode2utf(std::string& str, unsigned int c) {
#define ADD(str, c, flags, shift) \
	str.push_back((char) (flags | ((c >> shift) & 0x3F)))

	if (c < 0x80U)
		str.push_back((char) c);
	else if (c < 0x800U) {
		ADD(str, c, 0xC0, 6);
		ADD(str, c, 0x80, 0);
	}
	else if (c < 0x10000U) {
		ADD(str, c, 0xE0, 12);
		ADD(str, c, 0x80, 6);
		ADD(str, c, 0x80, 0);
	}
	else if (c < 0x200000U) {
		ADD(str, c, 0xF0, 18);
		ADD(str, c, 0x80, 12);
		ADD(str, c, 0x80, 6);
		ADD(str, c, 0x80, 0);
	}
	else if (c < 0x4000000U) {
		ADD(str, c, 0xF8, 24);
		ADD(str, c, 0x80, 18);
		ADD(str, c, 0x80, 12);
		ADD(str, c, 0x80, 6);
		ADD(str, c, 0x80, 0);
	}
	else if (c < 0x80000000U) {
		ADD(str, c, 0xFC, 30);
		ADD(str, c, 0x80, 24);
		ADD(str, c, 0x80, 18);
		ADD(str, c, 0x80, 12);
		ADD(str, c, 0x80, 6);
		ADD(str, c, 0x80, 0);
	}
#undef ADD
}

[toc] | [prev] | [next] | [standalone]

#9084

On 8/19/11 6:34 PM, Peter Poulsen wrote:

<cut/>

> I have tried google but it is extremely hard to find a good
> explanation of how to do it. Is there somebody that can give an
> example of a generic solution for my wchar2utf8() function?

A good starting point might be:

http://www.cl.cam.ac.uk/~mgk25/unicode.html
specially useful the links to 3rd party libs
http://www.cl.cam.ac.uk/~mgk25/unicode.html#libs

Hope it helps,
Giovanni

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.c++

csiph-web