Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!feeder.news-service.com!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path: <lis01361@izone.net.au>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.002
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'python': 0.07; 'accordingly': 0.09; 'only,': 0.09; 'streams.': 0.09; 'utf-8': 0.09; 'valueerror:': 0.09; 'pm,': 0.11; '>>>': 0.12; 'wrote:': 0.14; 'arg': 0.16; 'codec': 0.16; 'planes': 0.16; 'received:203.24': 0.16; 'subject:unicode': 0.16; 'true)': 0.16; '16,': 0.16; 'invalid': 0.16; 'traceback': 0.16; '(most': 0.16; 'bytes': 0.19; 'errors,': 0.19; 'cc:no real name:2**0': 0.20; 'cc:2**0': 0.20; 'header:In-Reply-To:1': 0.22; 'cc:addr:python- list': 0.22; 'right.': 0.22; 'thu,': 0.22; 'last):': 0.23; 'byte': 0.25; 'external': 0.29; 'messages.': 0.29; 'skip:" 30': 0.29; 'unicode': 0.29; 'cc:addr:python.org': 0.31; 'iso': 0.31; "can't": 0.31; 'skip:b 30': 0.31; '...': 0.32; 'file': 0.35; 'characters': 0.35; 'header:User-Agent:1': 0.35; '"",': 0.35; 'however': 0.37; 'agreed': 0.37; 'be,': 0.38; 'sequence': 0.38; 'no.': 0.38; 'so,': 0.38; 'could': 0.39; '2011': 0.62; 'upon': 0.63; 'below': 0.63; 'consortium': 0.68; 'reply-to:no real name:2**0': 0.72; 'header :Reply-To:1': 0.72; 'max,': 0.84; 'to:addr:charter.net': 0.84
In-Reply-To: <U8Lyp.1000$dL5.14@newsfe08.iad>
References: <OkDyp.2983$M61.450@newsfe07.iad> <mailman.1433.1305151801.9059.python-list@python.org> <vpEyp.981$dL5.736@newsfe08.iad> <mailman.1435.1305157329.9059.python-list@python.org> <KDGyp.180$0t1.7@newsfe04.iad> <mailman.1439.1305167541.9059.python-list@python.org> <874o50k1eb.fsf@benfinney.id.au> <U8Lyp.1000$dL5.14@newsfe08.iad>
Date: Thu, 12 May 2011 17:58:24 +1000
Subject: Re: unicode by default
From: "John Machin" <sjmachin@lexicon.net>
To: "harrismh777" <harrismh777@charter.net>
User-Agent: SquirrelMail/1.4.21
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Priority: 3 (Normal)
Importance: Normal
Cc: python-list@python.org
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.12
Precedence: list
Reply-To: sjmachin@lexicon.net
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.1450.1305187110.9059.python-list@python.org>
Lines: 34
NNTP-Posting-Host: 82.94.164.166
X-Trace: 1305187111 news.xs4all.nl 81476 [::ffff:82.94.164.166]:36295
X-Complaints-To: abuse@xs4all.nl
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:5216

On Thu, May 12, 2011 4:31 pm, harrismh777 wrote:

>
> So, the UTF-16 UTF-32 is INTERNAL only, for Python

NO. See one of my previous messages. UTF-16 and UTF-32, like UTF-8 are
encodings for the EXTERNAL representation of Unicode characters in byte
streams.

> I also was not aware that UTF-8 chars could be up to six(6) byes long
> from left to right.

It could be, once upon a time in ISO faerieland, when it was thought that
Unicode could grow to 2**32 codepoints. However ISO and the Unicode
consortium have agreed that 17 planes is the utter max, and accordingly a
valid UTF-8 byte sequence can be no longer than 4 bytes ... see below

    >>> chr(17 * 65536)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: chr() arg not in range(0x110000)
    >>> chr(17 * 65536 - 1)
    '\U0010ffff'
    >>> _.encode('utf8')
    b'\xf4\x8f\xbf\xbf'
    >>> b'\xf5\x8f\xbf\xbf'.decode('utf8')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "C:\python32\lib\encodings\utf_8.py", line 16, in decode
        return codecs.utf_8_decode(input, errors, True)
    UnicodeDecodeError: 'utf8' codec can't decode byte 0xf5 in position 0:
invalid start byte