Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed1.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.000 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'messages.': 0.04; 'output': 0.04; 'subject:Python': 0.05; 'encoded': 0.05; 'python3': 0.05; 'utf-8': 0.07; 'python': 0.09; 'charset': 0.09; 'encode': 0.09; 'eof': 0.09; 'gettext': 0.09; 'mkdir': 0.09; 'msgid': 0.09; 'skip:t 60': 0.09; 'subject:problems': 0.09; "system's": 0.09; 'bug': 0.10; 'code?': 0.16; 'codec': 0.16; 'detects': 0.16; 'folder,': 0.16; 'msgstr': 0.16; 'ordinal': 0.16; 'skip:{ 30': 0.16; 'subject:3.3': 0.16; 'subject:Unicode': 0.16; 'url:py': 0.16; 'skip:g 40': 0.17; 'skip:{ 20': 0.17; 'tries': 0.17; 'unicode': 0.17; 'thanks,': 0.18; 'module': 0.19; 'skip:" 30': 0.20; 'import': 0.21; 'downloaded': 0.24; 'linux': 0.24; 'skip:[ 10': 0.26; 'skip:" 20': 0.26; '(most': 0.27; 'skip:1 30': 0.27; 'translated': 0.27; 'message-id:@mail.gmail.com': 0.27; 'trouble': 0.28; 'run': 0.28; 'character': 0.29; 'included': 0.29; 'skip:& 10': 0.29; 'probably': 0.29; "i'm": 0.29; 'maybe': 0.29; 'skip:& 30': 0.30; 'url:python': 0.32; 'file': 0.32; 'skip:- 10': 0.32; 'getting': 0.33; 'skip:! 10': 0.33; 'traceback': 0.33; 'handle': 0.33; 'problem': 0.33; 'anyone': 0.33; 'to:addr:python- list': 0.33; 'skip:& 20': 0.33; "can't": 0.34; 'received:google.com': 0.34; 'follows:': 0.35; 'received:209.85.220': 0.35; 'received:209.85': 0.35; 'tool': 0.36; 'url:org': 0.36; 'skip:g 30': 0.36; 'skip:p 20': 0.36; 'correctly': 0.37; 'itself': 0.37; 'skip:t 40': 0.37; 'why': 0.37; 'received:209': 0.37; 'files': 0.38; 'skip:l 20': 0.38; 'to:addr:python.org': 0.39; 'skip:" 10': 0.40; 'header:Received:5': 0.40; 'skip:u 10': 0.60; 'subject:, ': 0.61; 'worth': 0.63; 'skip:c 50': 0.66; 'isolated': 0.84; 'url:cpython': 0.84 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=+H9YgbKLcyMbpTvqLeHYRqJr4xuAMWn/UkE7nqwu+g4=; b=dp1LfHZdzUfEPzpWgEcELb1TYq6hwsqFjniOdMYixWWFKKvsGH45fZJRBbNsnNsDdn FqYqK7SkgjVOmdVAHpAVBmIQxgBztNszPwja5E05jEV598a+K7+B7W+cYC85T3y1D8ej 1v0pFF61CznrZ/b9FASOxGK9P5oE6eSO5laGBVOnXh9afbFoJN4q2djlKH3er0Sx8hXW U6kNeiCORKqrO6iYvgKdkpIM8TEPeVv+AHzMtQcpSOQ1dQzSQIee4ikL2dot8py4MKB/ wr81y6Ths8inoAK5nqHSVfJo2kDSZ2tjSdQWwR09IIZ6exH4VTA1OavXupp/Qx6GP9x/ kA7w== MIME-Version: 1.0 Date: Sun, 30 Dec 2012 22:39:21 -0200 Subject: Python 3.3, gettext and Unicode problems From: Marcel Rodrigues To: python-list@python.org Content-Type: multipart/alternative; boundary=e89a8ff1c3e68ab5f104d21b3e4d X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 133 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1356914371 news.xs4all.nl 6934 [2001:888:2000:d::a6]:42227 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:35827 --e89a8ff1c3e68ab5f104d21b3e4d Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I'm using Python 3.3 (CPython) and am having trouble getting the standard gettext module to handle Unicode messages. My problem can be isolated as follows: I have 3 files in a folder: greeting.py, greeting.po and msgfmt.py. -- greeting.py -- import gettext t =3D gettext.translation("greeting", "locale", ["pt"]) _ =3D t.lgettext print("_charset =3D {0}\n".format(t._charset)) print(_("hello")) -- EOF -- -- greeting.po -- msgid "" msgstr "" "Project-Id-Version: 1.0\n" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=3DUTF-8\n" "Content-Transfer-Encoding: 8bit\n" msgid "hello" msgstr "ol=C3=A1" -- EOF -- msgfmt.py was downloaded from http://hg.python.org/cpython/file/9e6ead98762e/Tools/i18n/msgfmt.py, since this tool apparently isn't included in the python3 package available on Arch Linux official repositories. It's probably also worth noting that the file greeting.po is encoded itself as UTF-8. >From that folder, I run the following commands: $ mkdir -p locale/pt/LC_MESSAGES $ python msgfmt.py -o !$/greeting.mo greeting.po $ python greeting.py The output is: _charset =3D UTF-8 Traceback (most recent call last): File "greeting.py", line 7, in print(_("hello")) File "/usr/lib/python3.3/gettext.py", line 314, in lgettext return tmsg.encode(locale.getpreferredencoding()) UnicodeEncodeError: 'ascii' codec can't encode character '\xe1' in position 2: ordinal not in range(128) My interpretation of this output is that even though gettext correctly detects the MO file charset as UTF-8, it tries to encode the translated message with the system's "preferred encoding", which happens to be ASCII. Anyone know why this happens? Is this a bug on my code? Maybe I have misunderstood gettext... Thanks, Marcel --e89a8ff1c3e68ab5f104d21b3e4d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I'm using Python 3.3 (CPython) and am having trouble getting the standa= rd gettext module to handle Unicode messages.
My problem can be isolate= d as follows:

I have 3 files in a folder: greeting= .py, greeting.po and=C2=A0msgfmt.py.

-- greeting.py --
import gettext

t =3D get= text.translation("greeting", "locale", ["pt"]= )
_ =3D t.lgettext

print("_charset =3D {0}\n".format(t._cha= rset))
print(_("hello"))
-- EOF --

--=C2=A0greeting.po --
msgid ""
=
msgstr ""
= "Project-Id-Version: 1.0\n"=
"MIME-Version:= 1.0\n"
"Content-Type: text/plain; = charset=3DUTF-8\n"
"Content-Transfer-Encoding: 8bit\n"

msgid "hello&q= uot;
msgstr "ol= =C3=A1"
-- EOF --

msgfmt.py = was downloaded from=C2=A0http://hg.python.org/cpython/file/9e6ead98762= e/Tools/i18n/msgfmt.py, since this tool apparently isn't included i= n the python3 package available on Arch Linux official repositories.

It's probably also worth noting that the file=C2=A0= greeting.po is encoded itself as UTF-8.

From that = folder, I run the following commands:

$ mkdir -p locale/pt/LC_MESSAGES
$ python msgfmt.py -o !$/greetin= g.mo greeting.po
$ p= ython greeting.py

The output is:
_charset =3D UTF-8
<= div>
Traceback (most recent call last):
=
=C2=A0 File "greeting.py", = line 7, in <module>
=C2=A0 =C2=A0 print(_("hello"))
=C2=A0 File "/usr/lib/python3.3/gettext.py= ", line 314, in lgettext
=C2=A0 =C2=A0 return tmsg.encode= (locale.getpreferredencoding())
UnicodeEncodeError: 'ascii' codec can't encode char= acter '\xe1' in position 2: ordinal not in range(128)

My interpretation of this output is that even tho= ugh gettext correctly detects the MO file charset as UTF-8, it tries to enc= ode the translated message with the system's "preferred encoding&q= uot;, which happens to be ASCII.

Anyone know why this happens? Is this a bug on my code?= Maybe I have misunderstood gettext...

Thanks,

=C2=A0 Marcel
--e89a8ff1c3e68ab5f104d21b3e4d--