Path: csiph.com!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed2a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <52e33f8d$0$29999$c3e8da3$5496439d@news.astraweb.com>
References: <52e33f8d$0$29999$c3e8da3$5496439d@news.astraweb.com>
From: Oscar Benjamin <oscar.j.benjamin@gmail.com>
Date: Sat, 25 Jan 2014 21:15:12 +0000
Subject: Re: Trying to understand this moji-bake
To: "Steven D'Aprano" <steve+comp.lang.python@pearwood.info>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Cc: Python List <python-list@python.org>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.5981.1390684541.18130.python-list@python.org>
Lines: 90
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:64752

On 25 January 2014 04:37, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
>
> But using Python 2.7, I get a really bad case of moji-bake:
>
> [steve@ando ~]$ python2.7 -c "print u'=C3=B1=C3=B8=CE=BB=CF=80=D0=B9=D0=
=B6'"
> =C3=83=C2=B1=C3=83=C2=B8=C3=8E=C2=BB=C3=8F=C3=90=C2=B9=C3=90=C2=B6
>
> However, interactively it works fine:
>
> [steve@ando ~]$ python2.7 -E
> Python 2.7.2 (default, May 18 2012, 18:25:10)
> [GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> print u'=C3=B1=C3=B8=CE=BB=CF=80=D0=B9=D0=B6'
> =C3=B1=C3=B8=CE=BB=CF=80=D0=B9=D0=B6
>
> This occurs on at least two different machines, one using Centos and the
> other Debian.

Same for me. It's to do with using a u literal:

$ python2.7 -c "print('=C3=B1=C3=B8=CE=BB=CF=80=D0=B9=D0=B6')"
=C3=B1=C3=B8=CE=BB=CF=80=D0=B9=D0=B6
$ python2.7 -c "print(u'=C3=B1=C3=B8=CE=BB=CF=80=D0=B9=D0=B6')"
=C3=83=C2=B1=C3=83=C2=B8=C3=8E=C2=BB=C3=8F=E2=82=AC=C3=90=C2=B9=C3=90=C2=B6
$ python2.7 -c "print(repr('=C3=B1=C3=B8=CE=BB=CF=80=D0=B9=D0=B6'))"
'\xc3\xb1\xc3\xb8\xce\xbb\xcf\x80\xd0\xb9\xd0\xb6'
$ python2.7 -c "print(repr(u'=C3=B1=C3=B8=CE=BB=CF=80=D0=B9=D0=B6'))"
u'\xc3\xb1\xc3\xb8\xce\xbb\xcf\x80\xd0\xb9\xd0\xb6'

$ python2.7
Python 2.7.5+ (default, Sep 19 2013, 13:49:51)
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> b=3D'\xc3\xb1\xc3\xb8\xce\xbb\xcf\x80\xd0\xb9\xd0\xb6'
>>> print(b)
=C3=B1=C3=B8=CE=BB=CF=80=D0=B9=D0=B6
>>> s=3Du'\xc3\xb1\xc3\xb8\xce\xbb\xcf\x80\xd0\xb9\xd0\xb6'
>>> print(s)
=C3=83=C2=B1=C3=83=C2=B8=C3=8E=C2=BB=C3=8F=E2=82=AC=C3=90=C2=B9=C3=90=C2=B6
>>> print(s.encode('latin-1'))
=C3=B1=C3=B8=CE=BB=CF=80=D0=B9=D0=B6
>>> import sys
>>> sys.getdefaultencoding()
'ascii'

It works in the interactive prompt:

>>> s =3D '=C3=B1=C3=B8=CE=BB=CF=80=D0=B9=D0=B6'
>>> print(s)
=C3=B1=C3=B8=CE=BB=CF=80=D0=B9=D0=B6
>>> s =3D u'=C3=B1=C3=B8=CE=BB=CF=80=D0=B9=D0=B6'
>>> print(s)
=C3=B1=C3=B8=CE=BB=CF=80=D0=B9=D0=B6

But the interactive prompt has an associated encoding:

>>> import sys
>>> sys.stdout.encoding
'UTF-8'

If I put it into a utf-8 file with no encoding declared I get a SyntaxError=
:
$ cat tmp.py
s =3D u'=C3=B1=C3=B8=CE=BB=CF=80=D0=B9=D0=B6'
print(s)
oscar@tonis-laptop:~$ python2.7 tmp.py
  File "tmp.py", line 1
SyntaxError: Non-ASCII character '\xc3' in file tmp.py on line 1, but
no encoding declared; see http://www.python.org/peps/pep-0263.html for
details

If I add the encoding declaration it works:

oscar@tonis-laptop:~$ vim tmp.py
oscar@tonis-laptop:~$ cat tmp.py
# -*- coding: utf-8 -*-
s =3D u'=C3=B1=C3=B8=CE=BB=CF=80=D0=B9=D0=B6'
print(s)
oscar@tonis-laptop:~$ python2.7 tmp.py
=C3=B1=C3=B8=CE=BB=CF=80=D0=B9=D0=B6
oscar@tonis-laptop:~$

So I'd say that your original example should be a SyntaxError with
Python 2.7 but instead it implicitly uses latin-1.


Oscar