Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: Chris Angelico <rosuav@gmail.com>
Newsgroups: comp.lang.python
Subject: Re: Well, I finally ran into a Python Unicode problem, sort of
Date: Sun, 3 Jul 2016 17:41:59 +1000
Lines: 56
Message-ID: <mailman.36.1467531722.2295.python-list@python.org>
References: <0589521b-648a-47c1-8afa-6f746e0fb901@googlegroups.com> <CAPTjJmpXygTHwzMzPFtU96eanS_0GQvo1L4cYgEfqUa_HfGSOQ@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
In-Reply-To: <0589521b-648a-47c1-8afa-6f746e0fb901@googlegroups.com>
Precedence: list
Xref: csiph.com comp.lang.python:110983

On Sun, Jul 3, 2016 at 4:58 PM, John Ladasky <john_ladasky@sbcglobal.net> w=
rote:
> Up until today, every character I've tried has been accepted by the Pytho=
n interpreter as a legitimate character for inclusion in a variable name.  =
Now I'm copying a formula which defines a gradient.  The nabla symbol (=E2=
=88=87) is used in the naming of gradients.  Python isn't having it.  The i=
nterpreter throws a "SyntaxError: invalid character in identifier" when it =
encounters the =E2=88=87.
>
> I am now wondering what constitutes a valid character for an identifier, =
and how they were chosen.  Obviously, the Western alphabet and standard Gre=
ek letters work.  I just tried a few very weird characters from the Latin E=
xtended range, and some Cyrillic characters.  These are also fine.
>

Very good question! The detaily answer is here:

https://docs.python.org/3/reference/lexical_analysis.html#identifiers

> A philosophical question.  Why should any character be excluded from a va=
riable name, besides the fact that it might also be an operator?
>

In a way, that's exactly what's happening here. Python permits certain
categories of character as identifiers, leaving other categories
available for operators. Even though there aren't any non-ASCII
operators in a vanilla CPython, it's plausible that someone could
create a Python-based language with more operators (eg =E2=89=A0 NOT EQUAL =
TO
as an alternative to !=3D), and I'm sure you'd agree that saying "=E2=89=A0=
 =3D 1"
is nonsensical.

> This might be a problem I can solve, I'm not sure.  Is there a file that =
the Python interpreter refers to which defines the accepted variable name c=
haracters?  Perhaps I could just add =E2=88=87.
>

The key here is its Unicode category:

>>> unicodedata.category("=E2=88=87")
'Sm'

You could probably hack CPython to include Sm, and maybe Sc, Sk, and
So, as valid identifier characters. I'm not sure where, though, and
I've just spent a good bit of time delving (it's based on the
XID_Start and XID_Continue derived properties, but I have no idea
where they're defined - Tools/unicode/makeunicodedata.py looks
promising, but even there, I can't find it). And - or maybe instead -
you could appeal to the core devs to have the category/ies in question
added to the official Python spec. Symbols like that are a bit of a
grey area, so you may find that you're starting a huge debate :)

Have fun.

ChrisA