Re: unicode by default

Path	csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!feeder.news-service.com!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path	<lis01361@izone.net.au>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.002
X-Spam-Evidence	'H': 1.00; 'S': 0.00; 'python': 0.07; 'accordingly': 0.09; 'only,': 0.09; 'streams.': 0.09; 'utf-8': 0.09; 'valueerror:': 0.09; 'pm,': 0.11; '>>>': 0.12; 'wrote:': 0.14; 'arg': 0.16; 'codec': 0.16; 'planes': 0.16; 'received:203.24': 0.16; 'subject:unicode': 0.16; 'true)': 0.16; '16,': 0.16; 'invalid': 0.16; 'traceback': 0.16; '(most': 0.16; 'bytes': 0.19; 'errors,': 0.19; 'cc:no real name:20': 0.20; 'cc:20': 0.20; 'header:In-Reply-To:1': 0.22; 'cc:addr:python- list': 0.22; 'right.': 0.22; 'thu,': 0.22; 'last):': 0.23; 'byte': 0.25; 'external': 0.29; 'messages.': 0.29; 'skip:" 30': 0.29; 'unicode': 0.29; 'cc:addr:python.org': 0.31; 'iso': 0.31; "can't": 0.31; 'skip:b 30': 0.31; '...': 0.32; 'file': 0.35; 'characters': 0.35; 'header:User-Agent:1': 0.35; '"",': 0.35; 'however': 0.37; 'agreed': 0.37; 'be,': 0.38; 'sequence': 0.38; 'no.': 0.38; 'so,': 0.38; 'could': 0.39; '2011': 0.62; 'upon': 0.63; 'below': 0.63; 'consortium': 0.68; 'reply-to:no real name:2**0': 0.72; 'header :Reply-To:1': 0.72; 'max,': 0.84; 'to:addr:charter.net': 0.84
In-Reply-To	<U8Lyp.1000$dL5.14@newsfe08.iad>
References	<OkDyp.2983$M61.450@newsfe07.iad> <mailman.1433.1305151801.9059.python-list@python.org> <vpEyp.981$dL5.736@newsfe08.iad> <mailman.1435.1305157329.9059.python-list@python.org> <KDGyp.180$0t1.7@newsfe04.iad> <mailman.1439.1305167541.9059.python-list@python.org> <874o50k1eb.fsf@benfinney.id.au> <U8Lyp.1000$dL5.14@newsfe08.iad>
Date	Thu, 12 May 2011 17:58:24 +1000
Subject	Re: unicode by default
From	"John Machin" <sjmachin@lexicon.net>
To	"harrismh777" <harrismh777@charter.net>
User-Agent	SquirrelMail/1.4.21
MIME-Version	1.0
Content-Type	text/plain;charset=iso-8859-1
Content-Transfer-Encoding	8bit
X-Priority	3 (Normal)
Importance	Normal
Cc	python-list@python.org
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.12
Precedence	list
Reply-To	sjmachin@lexicon.net
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.1450.1305187110.9059.python-list@python.org> (permalink)
Lines	34
NNTP-Posting-Host	82.94.164.166
X-Trace	1305187111 news.xs4all.nl 81476 [::ffff:82.94.164.166]:36295
X-Complaints-To	abuse@xs4all.nl
Xref	x330-a1.tempe.blueboxinc.net comp.lang.python:5216

Show key headers only | View raw

On Thu, May 12, 2011 4:31 pm, harrismh777 wrote:

>
> So, the UTF-16 UTF-32 is INTERNAL only, for Python

NO. See one of my previous messages. UTF-16 and UTF-32, like UTF-8 are
encodings for the EXTERNAL representation of Unicode characters in byte
streams.

> I also was not aware that UTF-8 chars could be up to six(6) byes long
> from left to right.

It could be, once upon a time in ISO faerieland, when it was thought that
Unicode could grow to 2**32 codepoints. However ISO and the Unicode
consortium have agreed that 17 planes is the utter max, and accordingly a
valid UTF-8 byte sequence can be no longer than 4 bytes ... see below

    >>> chr(17 * 65536)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: chr() arg not in range(0x110000)
    >>> chr(17 * 65536 - 1)
    '\U0010ffff'
    >>> _.encode('utf8')
    b'\xf4\x8f\xbf\xbf'
    >>> b'\xf5\x8f\xbf\xbf'.decode('utf8')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "C:\python32\lib\encodings\utf_8.py", line 16, in decode
        return codecs.utf_8_decode(input, errors, True)
    UnicodeDecodeError: 'utf8' codec can't decode byte 0xf5 in position 0:
invalid start byte

Thread

unicode by default harrismh777 <harrismh777@charter.net> - 2011-05-11 16:37 -0500
  Re: unicode by default Ian Kelly <ian.g.kelly@gmail.com> - 2011-05-11 16:09 -0600
    Re: unicode by default harrismh777 <harrismh777@charter.net> - 2011-05-11 17:51 -0500
      Re: unicode by default "John Machin" <sjmachin@lexicon.net> - 2011-05-12 09:32 +1000
        Re: unicode by default harrismh777 <harrismh777@charter.net> - 2011-05-11 20:22 -0500
          Re: unicode by default MRAB <python@mrabarnett.plus.com> - 2011-05-12 03:31 +0100
            Re: unicode by default Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-05-12 03:16 +0000
              Re: unicode by default harrismh777 <harrismh777@charter.net> - 2011-05-11 22:44 -0500
                Re: unicode by default Terry Reedy <tjreedy@udel.edu> - 2011-05-12 00:12 -0400
                Re: unicode by default harrismh777 <harrismh777@charter.net> - 2011-05-12 01:43 -0500
                Re: unicode by default "John Machin" <sjmachin@lexicon.net> - 2011-05-12 14:14 +1000
                Re: unicode by default Benjamin Kaplan <benjamin.kaplan@case.edu> - 2011-05-11 21:14 -0700
                Re: unicode by default "John Machin" <sjmachin@lexicon.net> - 2011-05-12 14:41 +1000
                Re: unicode by default harrismh777 <harrismh777@charter.net> - 2011-05-12 01:14 -0500
                Re: unicode by default TheSaint <nobody@nowhere.net.no> - 2011-05-12 20:40 +0800
            Re: unicode by default Ben Finney <ben+python@benfinney.id.au> - 2011-05-12 14:07 +1000
              Re: unicode by default harrismh777 <harrismh777@charter.net> - 2011-05-12 01:31 -0500
                Re: unicode by default "John Machin" <sjmachin@lexicon.net> - 2011-05-12 17:58 +1000
                Re: unicode by default Ian Kelly <ian.g.kelly@gmail.com> - 2011-05-12 10:17 -0600
                Re: unicode by default jmfauth <wxjmfauth@gmail.com> - 2011-05-12 23:28 -0700
                Re: unicode by default harrismh777 <harrismh777@charter.net> - 2011-05-13 14:53 -0500
                Re: unicode by default Robert Kern <robert.kern@gmail.com> - 2011-05-13 15:18 -0500
                Re: unicode by default Terry Reedy <tjreedy@udel.edu> - 2011-05-13 21:41 -0400
                Re: unicode by default harrismh777 <harrismh777@charter.net> - 2011-05-14 02:41 -0500
                Re: unicode by default jmfauth <wxjmfauth@gmail.com> - 2011-05-14 03:26 -0700
                Re: unicode by default Terry Reedy <tjreedy@udel.edu> - 2011-05-14 16:26 -0400
                Re: unicode by default Ben Finney <ben+python@benfinney.id.au> - 2011-05-15 09:47 +1000
                Re: unicode by default Nobody <nobody@nowhere.com> - 2011-05-14 09:34 +0100
                Re: unicode by default Terry Reedy <tjreedy@udel.edu> - 2011-05-12 16:42 -0400
                Re: unicode by default Ian Kelly <ian.g.kelly@gmail.com> - 2011-05-12 16:25 -0600
          Re: unicode by default "John Machin" <sjmachin@lexicon.net> - 2011-05-12 13:54 +1000
  Re: unicode by default Benjamin Kaplan <benjamin.kaplan@case.edu> - 2011-05-11 15:34 -0700

csiph-web