Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Terry Reedy <tjreedy@udel.edu>
Subject: Re: Python 3.3, gettext and Unicode problems
Date: Sun, 30 Dec 2012 21:26:22 -0500
References: <CAFEv2m5dfn8StmwnoT8TTBTC=f_SMkijY4tAEOyFzq+RJ94_QQ@mail.gmail.com> <kbqqte$alg$1@ger.gmane.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Thunderbird/17.0
In-Reply-To: <kbqqte$alg$1@ger.gmane.org>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.1481.1356920828.29569.python-list@python.org>
Lines: 63
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:35829

On 12/30/2012 8:48 PM, Terry Reedy wrote:
> On 12/30/2012 7:39 PM, Marcel Rodrigues wrote:
>> I'm using Python 3.3 (CPython) and am having trouble getting the
>> standard gettext module to handle Unicode messages.

Addition to previous response.

>> import gettext
>>
>> t =3D gettext.translation("greeting", "locale", ["pt"])

Reading further, I see that this returns a GNUTranslations instance

>> _ =3D t.lgettext

So this calls its method:
'''
GNUTranslations.gettext(message)
Look up the message id in the catalog and return the corresponding=20
message string, as a Unicode string. If there is no entry in the catalog =

for the message id, and a fallback has been set, the look up is=20
forwarded to the fallback=E2=80=99s gettext() method. Otherwise, the mess=
age id=20
is returned.

GNUTranslations.lgettext(message)
Equivalent to gettext(), but the translation is returned as a bytestring =

encoded in the selected output charset, or in the preferred system=20
encoding if no encoding was explicitly set with set_output_charset().
'''
So if you want the unicode translation to be utf-8 encoded, either use=20
=2Egettext and encode it yourself, or use "t.set_output_charset('utf-8')"=
=20
to have it done automatically.

 >> print("_charset =3D {0}\n".format(t._charset))
 >> print(_("hello"))

But since you are printing to screen, I suggest using .gettext and let=20
print do the encoding to the screen encoding. If that still raises an=20
encoding error, then the problem is the console emulator. On windows,=20
for instance, IDLE windows handle the entire BMP charset while the=20
stupid Windows Command Prompt window does not (certainly not by default, =

and not yet, as far I know).

The encoding of the translations file on disk determines how the bytes=20
of the translation table should be *decoded when read, to create unicode =

strings. It does not determine how those strings should be *encoded*=20
when sent to a particular destination. That may depend on the=20
destination. Multilingual international sites used to encode pages in=20
different limited national encodings, according to the language and=20
destination. Now many encode and send *everything* as utf-8. I think=20
this is the proper policy now. .lgettext seems oriented to the older,=20
pre utf-8, national locale encoding way of doing things.

--=20
Terry Jan Reedy