Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed1a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Fri, 01 Aug 2014 23:18:47 -0700
From: Glenn Linderman <v+python@g.nevcal.com>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: Python 3 is killing Python
References: <mailman.10416.1401304999.18130.python-list@python.org> <57ajo9poljjre4c4ig0n0ss8kph8k78lp0@4ax.com> <5389cb53$0$29978$c3e8da3$5496439d@news.astraweb.com> <99b7b2a2-7521-42d7-a5a0-1a35d4d5b922@googlegroups.com> <53C4A454.9010600@gmail.com> <mailman.11826.1405398039.18130.python-list@python.org> <lq365f$4c9$1@speranza.aioe.org> <mailman.11836.1405429260.18130.python-list@python.org> <lq3bvu$ob4$1@dont-email.me> <mailman.11838.1405434699.18130.python-list@python.org> <87zjga4j4v.fsf@elektro.pacujo.net> <53c57bae$0$9505$c3e8da3$5496439d@news.astraweb.com> <87iomy4ciy.fsf@elektro.pacujo.net> <53c5f6dc$0$9505$c3e8da3$5496439d@news.astraweb.com> <87egxl4zq8.fsf@elektro.pacujo.net> <lq6255$gi5$1@ger.gmane.org>
In-Reply-To: <lq6255$gi5$1@ger.gmane.org>
Content-Type: multipart/alternative; boundary="------------020102020609010206010903"
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.12534.1406960354.18130.python-list@python.org>
Lines: 108
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:75515

This is a multi-part message in MIME format.
--------------020102020609010206010903
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 7/16/2014 7:27 AM, Frank Millman wrote:
> I just tried an experiment in my own project. Ned Batchelder, in his
> Pragmatic Unicode presentation, http://nedbatchelder.com/text/unipain.html,
> suggests that you always have some unicode characters in your data, just to
> ensure that they are handled correctly. He has a tongue-in-cheek example
> which spells the word PYTHON using various exotic unicode characters. I used
> this to populate a field in my database, to see if it would display in my
> browser-based client.
>
> The hardest part was getting it in. There are 6 characters, but utf-8
> requires 16 bytes to store it -
>
>      b'\xe2\x84\x99\xc6\xb4\xe2\x98\x82\xe2\x84\x8c\xc3\xb8\xe1\xbc\xa4'.decode('utf-8')
>
> However, that was it. Without any changes to my program, it read it from the
> database and displayed it on the screen. IE8 could only display 2 out of the
> 6 characters correctly, and Chrome could display 5 out of 6, but that is a
> separate issue. Python3 handled it perfectly.

wrapping the above in a print(), on Windows, I get:

Traceback (most recent call last):
   File "D:\my\py\python-utf8.py", line 1, in <module>
print(b'\xe2\x84\x99\xc6\xb4\xe2\x98\x82\xe2\x84\x8c\xc3\xb8\xe1\xbc\xa4'.decode('utf-8'))
   File "C:\Python33\lib\encodings\cp437.py", line 19, in encode
     return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 
0-5: character maps to <undefined>

So Python3 doesn't handle it perfectly on Windows.  And I saw someone 
blame the Windows console for that... but the Windows console can 
properly display all those characters if the proper APIs are used. The 
bug is 7 years old: http://bugs.python.org/issue1602 and hasn't been 
fixed, although the technology for fixing it is available, and various 
workarounds (with limitations) have been available for 5 years, and 
patches have been available for 3 years that work pretty good. However, 
just a few days ago, 26 July 2014, Drekin had an insight that may 
possibly lead to a patch that will work well enough to be integrated 
into some future version of Python... I hope he follows up on it. This 
is a serious limitation, and it is, and always has been, a bug in Python 
3 Unicode handling on Windows.

--------------020102020609010206010903
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#330033">
    <div class="moz-cite-prefix">On 7/16/2014 7:27 AM, Frank Millman
      wrote:<br>
    </div>
    <blockquote cite="mid:lq6255$gi5$1@ger.gmane.org" type="cite">
      <pre wrap="">I just tried an experiment in my own project. Ned Batchelder, in his 
Pragmatic Unicode presentation, <a class="moz-txt-link-freetext" href="http://nedbatchelder.com/text/unipain.html">http://nedbatchelder.com/text/unipain.html</a>, 
suggests that you always have some unicode characters in your data, just to 
ensure that they are handled correctly. He has a tongue-in-cheek example 
which spells the word PYTHON using various exotic unicode characters. I used 
this to populate a field in my database, to see if it would display in my 
browser-based client.

The hardest part was getting it in. There are 6 characters, but utf-8 
requires 16 bytes to store it -

    b'\xe2\x84\x99\xc6\xb4\xe2\x98\x82\xe2\x84\x8c\xc3\xb8\xe1\xbc\xa4'.decode('utf-8')

However, that was it. Without any changes to my program, it read it from the 
database and displayed it on the screen. IE8 could only display 2 out of the 
6 characters correctly, and Chrome could display 5 out of 6, but that is a 
separate issue. Python3 handled it perfectly.
</pre>
    </blockquote>
    <br>
    wrapping the above in a print(), on Windows, I get:<br>
    <br>
    Traceback (most recent call last):<br>
      File "D:\my\py\python-utf8.py", line 1, in &lt;module&gt;<br>
       
print(b'\xe2\x84\x99\xc6\xb4\xe2\x98\x82\xe2\x84\x8c\xc3\xb8\xe1\xbc\xa4'.decode('utf-8'))<br>
      File "C:\Python33\lib\encodings\cp437.py", line 19, in encode<br>
        return codecs.charmap_encode(input,self.errors,encoding_map)[0]<br>
    UnicodeEncodeError: 'charmap' codec can't encode characters in
    position 0-5: character maps to &lt;undefined&gt;<br>
    <br>
    So Python3 doesn't handle it perfectly on Windows.  And I saw
    someone blame the Windows console for that... but the Windows
    console can properly display all those characters if the proper APIs
    are used. The bug is 7 years old: <a class="moz-txt-link-freetext" href="http://bugs.python.org/issue1602">http://bugs.python.org/issue1602</a>
    and hasn't been fixed, although the technology for fixing it is
    available, and various workarounds (with limitations) have been
    available for 5 years, and patches have been available for 3 years
    that work pretty good. However, just a few days ago, 26 July 2014,
    Drekin had an insight that may possibly lead to a patch that will
    work well enough to be integrated into some future version of
    Python... I hope he follows up on it. This is a serious limitation,
    and it is, and always has been, a bug in Python 3 Unicode handling
    on Windows.<br>
  </body>
</html>

--------------020102020609010206010903--