Path: csiph.com!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder1.enfer-du-nord.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Newsgroups: comp.lang.python
Date: Thu, 8 Nov 2012 11:30:37 -0800 (PST)
In-Reply-To: <mailman.3457.1352399533.27098.python-list@python.org>
Complaints-To: groups-abuse@google.com
Injection-Info: glegroupsg2000goo.googlegroups.com; posting-host=83.79.110.133; posting-account=ung4FAoAAAC46zhHJ0Nsnuox7M5gDvs_
References: <09a3d20b-5871-47f4-9218-df119698e405@m4g2000yqf.googlegroups.com> <CAHVvXxT8SpK24HtsS8XTfs0CHTBofh3=deYw5xY1nq5ctATjrg@mail.gmail.com> <509AF3EF.8050108@gmail.com> <CAHVvXxSg0XWjZmA1pnEvBB_b2T9pS91YdjhF2duZHmBuLbXNAw@mail.gmail.com> <mailman.3436.1352383603.27098.python-list@python.org> <65910cea-f145-409c-a579-9f0cda499546@googlegroups.com> <mailman.3457.1352399533.27098.python-list@python.org>
User-Agent: G2/1.0
MIME-Version: 1.0
Subject: Re: Right solution to unicode error?
From: wxjmfauth@gmail.com
To: comp.lang.python@googlegroups.com
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Cc: python-list@python.org, wxjmfauth@gmail.com
Precedence: list
Message-ID: <mailman.3461.1352403047.27098.python-list@python.org>
Lines: 247
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:32975

Le jeudi 8 novembre 2012 19:32:14 UTC+1, Oscar Benjamin a =C3=A9crit=C2=A0:
> On 8 November 2012 15:05,  <wxjmfauth@gmail.com> wrote:
>=20
> > Le jeudi 8 novembre 2012 15:07:23 UTC+1, Oscar Benjamin a =C3=A9crit :
>=20
> >> On 8 November 2012 00:44, Oscar Benjamin <oscar.j.benjamin@gmail.com> =
wrote:
>=20
> >> > On 7 November 2012 23:51, Andrew Berg <bahamutzero8825@gmail.com> wr=
ote:
>=20
> >> >> On 2012.11.07 17:27, Oscar Benjamin wrote:
>=20
> >>
>=20
> >> >>> Are you using cmd.exe (standard Windows terminal)? If so, it does =
not
>=20
> >> >>> support unicode
>=20
> >>
>=20
> >> >> Actually, it does. Code page 65001 is UTF-8. I know that doesn't he=
lp
>=20
> >> >> the OP since Python versions below 3.3 don't support cp65001, but I
>=20
> >> >> think it's important to point out that the Windows command line sys=
tem
>=20
> >> >> (it is not unique to cmd) does in fact support Unicode.
>=20
> >>
>=20
> >> > I have tried to use code page 65001 and it didn't work for me even i=
f
>=20
> >> > I did use a version of Python (possibly 3.3 alpha) that claimed to
>=20
> >> > support it.
>=20
> >>
>=20
> >> I stand corrected. I've just checked and codepage 65001 does work in
>=20
> >> cmd.exe (on this machine):
>=20
> >>
>=20
> >> O:\>chcp 65001
>=20
> >> Active code page: 65001
>=20
> >>
>=20
> >> O:\>Q:\tools\Python33\python -c print('abc\u2013def')
>=20
> >> abc-def
>=20
> >>
>=20
> >> O:\>Q:\tools\Python33\python -c print('\u03b1')
>=20
> >> =CE=B1
>=20
> >>
>=20
> >> It would be a lot better though if it just worked straight away
>=20
> >> without me needing to set the code page (like the terminal in every
>=20
> >> other OS I use).
>=20
> >
>=20
> > It *WORKS* straight away. The problem is that
>=20
> > people do not wish to use unicode correctly
>=20
> > (eg. Mulder's example).
>=20
> > Read the point 1) and 4) in my previous post.
>=20
> >
>=20
> > Unicode and in general the coding of the characters
>=20
> > have nothing to do with the os's or programming languages.
>=20
>=20
>=20
> I don't know what you mean that it works "straight away".
>=20
>=20
>=20
> The default code page on my machine is cp850.
>=20
>=20
>=20
> O:\>chcp
>=20
> Active code page: 850
>=20
>=20
>=20
> cp850 doesn't understand utf-8. It just prints garbage:
>=20
>=20
>=20
> O:\>Q:\tools\Python33\python -c "import sys;
>=20
> sys.stdout.buffer.write('\u03b1\n'.encode('utf-8'))"
>=20
> =E2=95=AC=E2=96=92
>=20
>=20
>=20
> Using the correct encoding doesn't help:
>=20
>=20
>=20
> O:\>Q:\tools\Python33\python -c "import sys;
>=20
> sys.stdout.buffer.write('\u03b1\n'.encode('cp850'))"
>=20
> Traceback (most recent call last):
>=20
>   File "<string>", line 1, in <module>
>=20
>   File "Q:\tools\Python33\lib\encodings\cp850.py", line 12, in encode
>=20
>     return codecs.charmap_encode(input,errors,encoding_map)
>=20
> UnicodeEncodeError: 'charmap' codec can't encode character '\u03b1' in
>=20
> position 0: character maps to
>=20
>  <undefined>
>=20
>=20
>=20
> O:\>Q:\tools\Python33\python -c "import sys;
>=20
> sys.stdout.buffer.write('\u03b1\n'.encode(sys.stdout.en
>=20
> coding))"
>=20
> Traceback (most recent call last):
>=20
>   File "<string>", line 1, in <module>
>=20
>   File "Q:\tools\Python33\lib\encodings\cp850.py", line 12, in encode
>=20
>     return codecs.charmap_encode(input,errors,encoding_map)
>=20
> UnicodeEncodeError: 'charmap' codec can't encode character '\u03b1' in
>=20
> position 0: character maps to
>=20
>  <undefined>
>=20
>=20
>=20
> If I want the other characters to work I need to change the code page:
>=20
>=20
>=20
> O:\>chcp 65001
>=20
> Active code page: 65001
>=20
>=20
>=20
> O:\>Q:\tools\Python33\python -c "import sys;
>=20
> sys.stdout.buffer.write('\u03b1\n'.encode('utf-8'))"
>=20
> =CE=B1
>=20
>=20
>=20
> O:\>Q:\tools\Python33\python -c "import sys;
>=20
> sys.stdout.buffer.write('\u03b1\n'.encode(sys.stdout.en
>=20
> coding))"
>=20
> =CE=B1
>=20
>=20
>=20
>=20
>=20
> Oscar

You are confusing two things. The coding of the
characters and the set of the characters (glyphes/graphemes)
of a coding scheme.

It is always possible to encode safely an unicode, but
the target coding may not contain the character.

Take a look at the output of this "special" interactive
interpreter" where the host coding (sys.stdout.encoding)
can be change on the fly.


>>> s =3D '=C3=A9l=C3=A9phant\u2013abc=C3=A9=C5=93=E2=82=AC'
>>> sys.stdout.encoding
'<unicode>'
>>> s
'=C3=A9l=C3=A9phant=E2=80=93abc=C3=A9=C5=93=E2=82=AC'
>>>=20
>>> sys.stdout.encoding =3D 'cp1252'
>>> s.encode('cp1252')
'=C3=A9l=C3=A9phant=E2=80=93abc=C3=A9=C5=93=E2=82=AC'
>>> sys.stdout.encoding =3D 'cp850'
>>> s.encode('cp850')
Traceback (most recent call last):
  File "<eta last command>", line 1, in <module>
  File "C:\Python32\lib\encodings\cp850.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character '\u2013'
in position 8: character maps to <undefined>
>>> # but
>>> s.encode('cp850', 'replace')
'=C3=A9l=C3=A9phant?abc=C3=A9??'
>>>=20
>>> sys.stdout.encoding =3D 'utf-8'
>>> s
'=C3=83=C2=A9l=C3=83=C2=A9phant=C3=A2=E2=82=AC=E2=80=9Cabc=C3=83=C2=A9=C3=
=85=E2=80=9C=C3=A2=E2=80=9A=C2=AC'
>>> s.encode('utf-8')
'=C3=A9l=C3=A9phant=E2=80=93abc=C3=A9=C5=93=E2=82=AC'
>>>=20
>>> sys.stdout.encoding =3D 'utf-16-le'  <<<<<<<<<
>>> s
' =C3=A9 l =C3=A9 p h a n t =13 a b c =C3=A9 S =C2=AC '
>>> s.encode('utf-16-le')
'=C3=A9l=C3=A9phant=E2=80=93abc=C3=A9=C5=93=E2=82=AC'

<<<<<<<<<<< some cheating here do to the mail system, it really looks like =
this.

jmf