Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!news.stack.nl!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <87a99q5a08.fsf@elektro.pacujo.net>
References: <mailman.10509.1401552642.18130.python-list@python.org> <CAN8CLgk5y0Of35RzEhm2-3OfyyzMqUfLABX7xAY-gQKqapYrJA@mail.gmail.com> <loom.20140602T094123-717@post.gmane.org> <CAN8CLgn_fXb1mqt3bZzeRw0d6TC=UykK5n0HTbA3JKHFo8gEHg@mail.gmail.com> <538C5BB8.1020702@chamonix.reportlab.co.uk> <mailman.10575.1401744891.18130.python-list@python.org> <bv540tFca0mU1@mid.individual.net> <lmjphk$bl7$1@ger.gmane.org> <mailman.10625.1401805111.18130.python-list@python.org> <538f1a61$0$29978$c3e8da3$5496439d@news.astraweb.com> <bva1ccFdr03U1@mid.individual.net> <53902bb1$0$11109$c3e8da3@news.astraweb.com> <87wqcvu20h.fsf@elektro.pacujo.net> <7b3543f6-6f62-49c5-abdc-e2783fd6d629@googlegroups.com> <87oay7tnxt.fsf@elektro.pacujo.net> <mailman.10755.1401991925.18130.python-list@python.org> <87tx7z5hvw.fsf@elektro.pacujo.net> <mailman.10815.1402062359.18130.python-list@python.org> <87egz25dsd.fsf@elektro.pacujo.net> <mailman.10818.1402066977.18130.python-list@python.org> <87a99q5a08.fsf@elektro.pacujo.net>
Date: Sat, 7 Jun 2014 01:50:50 +1000
Subject: Re: Python 3.2 has some deadly infection
From: Chris Angelico <rosuav@gmail.com>
Cc: "python-list@python.org" <python-list@python.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.10820.1402069852.18130.python-list@python.org>
Lines: 50
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:72857

On Sat, Jun 7, 2014 at 1:32 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Michael Torrie <torriem@gmail.com>:
>
>> On 06/06/2014 08:10 AM, Marko Rauhamaa wrote:
>>> Ethan Furman <ethan@stoneleaf.us>:
>>>> ASCII is *not* the state of "this string has no encoding" -- that
>>>> would be Unicode; a Unicode string, as a data type, has no encoding.
>>>
>>> Huh?
>>
>> [...]
>>
>> What part of his statement are you saying "Huh?" about?
>
> Unicode, like ASCII, is a code. Representing text in unicode is
> encoding.

Yes and no. "ASCII" means two things: Firstly, it's a mapping from the
letter A to the number 65, from the exclamation mark to 33, from the
backslash to 92, and so on. And secondly, it's an encoding of those
numbers into the lowest seven bits of a byte, with the high byte left
clear. Between those two, you get a means of representing the letter
'A' as the byte 0x41, and one of them is an encoding.

"Unicode", on the other hand, is only the first part. It maps all the
same characters to the same numbers that ASCII does, and then adds a
few more... a few followed by a few, followed by... okay, quite a lot
more. Unicode specifies that the character OK HAND SIGN, which looks
like =F0=9F=91=8C if you have the right font, is number 1F44C in hex (12807=
6
decimal). This is the "Universal Character Set" or UCS.

ASCII could specify a single encoding, because that encoding makes
sense for nearly all purposes. (There are times when you transmit
ASCII text and use the high bit to mean something else, like parity or
"this is the end of a word" or something, but even then, you follow
the same convention of packing a number into the low seven bits of a
byte.) Unicode can't, because there are many different pros and cons
to the different encodings, and so we have UCS Transformation Formats
like UTF-8 and UTF-32. Each one is an encoding that maps a codepoint
to a sequence of bytes.

You can't represent text in "Unicode" in a computer. Somewhere along
the way, you have to figure out how to store those codepoints as
bytes, or something more concrete (you could, for instance, use a
Python list of Python integers; I can't say that it would be in any
way more efficient than alternatives, but it would be plausible); and
that's the encoding.

ChrisA