Path: csiph.com!usenet.pasdenom.info!news.redatomik.org!newsfeed.xs4all.nl!newsfeed1a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <CA+gt_a82WGXHUZhcdbTUWG+TRV1Ys1ZSrkGjOxgavZGjAh9FiQ@mail.gmail.com>
References: <CA+gt_a82WGXHUZhcdbTUWG+TRV1Ys1ZSrkGjOxgavZGjAh9FiQ@mail.gmail.com>
Date: Mon, 11 May 2015 02:06:56 +1000
Subject: Re: Why does unicode-escape decode escape symbols that are already escaped?
From: Chris Angelico <rosuav@gmail.com>
Cc: "python-list@python.org" <python-list@python.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.313.1431274025.12865.python-list@python.org>
Lines: 43
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:90292

On Mon, May 11, 2015 at 1:53 AM, Somelauw . <somelauw@gmail.com> wrote:
> In Python 3, decoding "=E2=82=AC" with unicode-escape returns '=C3=A2\x82=
=C2=AC' which in my
> opinion doesn't make sense.
> The =E2=82=AC already is decoded; if it were encoded it would look like t=
his:
> '\u20ac'.
> So why is it doing this?
>
> In Python 2 the behaviour is similar, but slightly different.
>
> $ python3 -S
> Python 3.3.3 (default, Nov 27 2013, 17:12:35)
> [GCC 4.8.2] on linux
>>>> import codecs
>>>> codecs.decode('=E2=82=AC', 'unicode-escape')
> '=C3=A2\x82=C2=AC'
>>>> codecs.encode('=E2=82=AC', 'unicode-escape')
> b'\\u20ac'
>>>>

Whenever you start encoding and decoding, you need to know whether
you're working with bytes->text, text->bytes, or something else. In
the case of unicode-escape, it expects to encode text into bytes, as
you can see with your second example - you give it a Unicode string,
and get back a byte string. When you attempt to *decode* a Unicode
string, that doesn't actually make sense, so it first gets *encoded*
to bytes, before being decoded. What you're actually seeing there is
that the one-character string is being encoded into a three-byte UTF-8
sequence,and then the unicode-escape decode takes those bytes and
interprets them as characters; as it happens, that's equivalent to a
Latin-1 decode:

>>> '=E2=82=AC'.encode('utf-8').decode('latin-1')
'=C3=A2\x82=C2=AC'

I strongly suggest leaving the codecs module aside, and working
exclusively with the str.encode() and bytes.decode() methods, if you
possibly can. If you can't, at very least keep track in your head of
what is text and what is bytes, and which way things change in every
transformation.

ChrisA