Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!news.dougwise.org!nntpfeed.proxad.net!proxad.net!feeder1-2.proxad.net!news.wiretrip.org!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!newsgate.cistron.nl!newsgate.news.xs4all.nl!194.109.133.85.MISMATCH!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; b=dATA697NFD8pAoIidurPbRwixf9NCtjhtU8GZMRJ4YNU1a/VmJRq7HIE+xrTfXmSdl 4Ls12gZI+qxJ+mzyMTLhb93g+CZigRMoGMF/+C7w7WQYTIZ+OmPTvtmn/FyPX+80FzwE LuzM0rzOq/ls471WEabdAhFrAQacvszBhxujM=
MIME-Version: 1.0
In-Reply-To: <OkDyp.2983$M61.450@newsfe07.iad>
References: <OkDyp.2983$M61.450@newsfe07.iad>
From: Ian Kelly <ian.g.kelly@gmail.com>
Date: Wed, 11 May 2011 16:09:29 -0600
Subject: Re: unicode by default
To: Python <python-list@python.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.1433.1305151801.9059.python-list@python.org>
Lines: 58
NNTP-Posting-Host: 82.94.164.166
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:5168

On Wed, May 11, 2011 at 3:37 PM, harrismh777 <harrismh777@charter.net> wrot=
e:
> hi folks,
> =A0 I am puzzled by unicode generally, and within the context of python
> specifically. For one thing, what do we mean that unicode is used in pyth=
on
> 3.x by default. (I know what default means, I mean, what changed?)

The `unicode' class was renamed to `str', and a stripped-down version
of the 2.X `str' class was renamed to `bytes'.

> =A0 I think part of my problem is that I'm spoiled (American, ascii herit=
age)
> and have been either stuck in ascii knowingly, or UTF-8 without knowing
> (just because the code points lined up). I am confused by the implication=
s
> for using 3.x, because I am reading that there are significant things to =
be
> aware of... what?

Mainly Python 3 no longer does explicit conversion between bytes and
unicode, requiring the programmer to be explicit about such
conversions.  If you have Python 2 code that is sloppy about this, you
may get some Unicode encode/decode errors when trying to run the same
code in Python 3.  The 2to3 tool can help somewhat with this, but it
can't prevent all problems.

> =A0 On my installation 2.6 =A0sys.maxunicode comes up with 1114111, and m=
y 2.7
> and 3.2 installs come up with 65535 each. So, I am assuming that 2.6 was
> compiled with UCS-4 (UTF-32) option for 4 byte unicode(?) and that the
> default compile option for 2.7 & 3.2 (I didn't change anything) is set fo=
r
> UCS-2 (UTF-16) or 2 byte unicode(?). =A0 Do I understand this much correc=
tly?

I think that UCS-2 has always been the default unicode width for
CPython, although the exact representation used internally is an
implementation detail.

> =A0 The books say that the .py sources are UTF-8 by default... and that 3=
.x is
> either UCS-2 or UCS-4. =A0If I use the file handling capabilities of Pyth=
on in
> 3.x (by default) what encoding will be used, and how will that affect the
> output?

If you open a file in binary mode, the result is a non-decoded byte stream.

If you open a file in text mode and do not specify an encoding, then
the result of locale.getpreferredencoding() is used for decoding, and
the result is a unicode stream.

> =A0 If I do not specify any code points above ascii 0xFF does any of this
> matter anyway?

You mean 0x7F, and probably, due to the need to explicitly encode and decod=
e.