Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <65910cea-f145-409c-a579-9f0cda499546@googlegroups.com>
References: <09a3d20b-5871-47f4-9218-df119698e405@m4g2000yqf.googlegroups.com> <CAHVvXxT8SpK24HtsS8XTfs0CHTBofh3=deYw5xY1nq5ctATjrg@mail.gmail.com> <509AF3EF.8050108@gmail.com> <CAHVvXxSg0XWjZmA1pnEvBB_b2T9pS91YdjhF2duZHmBuLbXNAw@mail.gmail.com> <mailman.3436.1352383603.27098.python-list@python.org> <65910cea-f145-409c-a579-9f0cda499546@googlegroups.com>
Date: Thu, 8 Nov 2012 18:32:11 +0000
Subject: Re: Right solution to unicode error?
From: Oscar Benjamin <oscar.j.benjamin@gmail.com>
To: wxjmfauth@gmail.com
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Cc: python-list@python.org
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.3457.1352399533.27098.python-list@python.org>
Lines: 98
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:32970

On 8 November 2012 15:05,  <wxjmfauth@gmail.com> wrote:
> Le jeudi 8 novembre 2012 15:07:23 UTC+1, Oscar Benjamin a =C3=A9crit :
>> On 8 November 2012 00:44, Oscar Benjamin <oscar.j.benjamin@gmail.com> wr=
ote:
>> > On 7 November 2012 23:51, Andrew Berg <bahamutzero8825@gmail.com> wrot=
e:
>> >> On 2012.11.07 17:27, Oscar Benjamin wrote:
>>
>> >>> Are you using cmd.exe (standard Windows terminal)? If so, it does no=
t
>> >>> support unicode
>>
>> >> Actually, it does. Code page 65001 is UTF-8. I know that doesn't help
>> >> the OP since Python versions below 3.3 don't support cp65001, but I
>> >> think it's important to point out that the Windows command line syste=
m
>> >> (it is not unique to cmd) does in fact support Unicode.
>>
>> > I have tried to use code page 65001 and it didn't work for me even if
>> > I did use a version of Python (possibly 3.3 alpha) that claimed to
>> > support it.
>>
>> I stand corrected. I've just checked and codepage 65001 does work in
>> cmd.exe (on this machine):
>>
>> O:\>chcp 65001
>> Active code page: 65001
>>
>> O:\>Q:\tools\Python33\python -c print('abc\u2013def')
>> abc-def
>>
>> O:\>Q:\tools\Python33\python -c print('\u03b1')
>> =CE=B1
>>
>> It would be a lot better though if it just worked straight away
>> without me needing to set the code page (like the terminal in every
>> other OS I use).
>
> It *WORKS* straight away. The problem is that
> people do not wish to use unicode correctly
> (eg. Mulder's example).
> Read the point 1) and 4) in my previous post.
>
> Unicode and in general the coding of the characters
> have nothing to do with the os's or programming languages.

I don't know what you mean that it works "straight away".

The default code page on my machine is cp850.

O:\>chcp
Active code page: 850

cp850 doesn't understand utf-8. It just prints garbage:

O:\>Q:\tools\Python33\python -c "import sys;
sys.stdout.buffer.write('\u03b1\n'.encode('utf-8'))"
=E2=95=AC=E2=96=92

Using the correct encoding doesn't help:

O:\>Q:\tools\Python33\python -c "import sys;
sys.stdout.buffer.write('\u03b1\n'.encode('cp850'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "Q:\tools\Python33\lib\encodings\cp850.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character '\u03b1' in
position 0: character maps to
 <undefined>

O:\>Q:\tools\Python33\python -c "import sys;
sys.stdout.buffer.write('\u03b1\n'.encode(sys.stdout.en
coding))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "Q:\tools\Python33\lib\encodings\cp850.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character '\u03b1' in
position 0: character maps to
 <undefined>

If I want the other characters to work I need to change the code page:

O:\>chcp 65001
Active code page: 65001

O:\>Q:\tools\Python33\python -c "import sys;
sys.stdout.buffer.write('\u03b1\n'.encode('utf-8'))"
=CE=B1

O:\>Q:\tools\Python33\python -c "import sys;
sys.stdout.buffer.write('\u03b1\n'.encode(sys.stdout.en
coding))"
=CE=B1


Oscar