Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: Robert Kern <robert.kern@gmail.com>
Newsgroups: comp.lang.python
Subject: Re: Well, I finally ran into a Python Unicode problem, sort of
Date: Sun, 3 Jul 2016 10:16:59 +0100
Lines: 18
Message-ID: <mailman.37.1467537428.2295.python-list@python.org>
References: <0589521b-648a-47c1-8afa-6f746e0fb901@googlegroups.com> <lf57fd341sr.fsf@ling.helsinki.fi> <nlal6c$rng$1@ger.gmane.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Trace: news.uni-berlin.de c5aOl9CAJojFaSOn+IBDmwipd4gc/MipFRlxzgEZ77uQ==
Return-Path: <python-python-list@m.gmane.org>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.000
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'subject:Python': 0.05; 'attempted': 0.09; 'finite': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'subject:into': 0.09; 'underlying': 0.09; 'python': 0.10; 'explicitly': 0.15; 'catch-all': 0.16; 'identifier.': 0.16; 'kern': 0.16; 'literal,': 0.16; 'operators,': 0.16; 'received:80.91.229.3': 0.16; 'received:io': 0.16; 'received:plane.gmane.org': 0.16; 'received:psf.io': 0.16; 'subject:Unicode': 0.16; 'wrote:': 0.16; 'numerical': 0.18; 'interpret': 0.22; 'keywords,': 0.22; 'occurs': 0.22; 'subject:problem': 0.22; 'seems': 0.23; 'header:In-Reply-To:1': 0.24; 'header:User-Agent:1': 0.26; 'header:X-Complaints-To:1': 0.26; 'error': 0.27; 'character': 0.29; 'comments': 0.30; 'guess': 0.31; 'could': 0.35; 'attempt': 0.35; 'lists.': 0.35; 'robert': 0.35; 'but': 0.36; 'to:addr:python-list': 0.36; 'subject:: ': 0.37; 'received:org': 0.37; 'to:addr:python.org': 0.40; 'default': 0.61; 'our': 0.64; 'world': 0.64; 'believe': 0.66; 'eco': 0.84; 'terrible': 0.84; 'received:86': 0.93
X-Injected-Via-Gmane: http://gmane.org/
X-Gmane-NNTP-Posting-Host: cpc91458-cmbg20-2-0-cust337.5-4.cable.virginm.net
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.2.0
In-Reply-To: <lf57fd341sr.fsf@ling.helsinki.fi>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <nlal6c$rng$1@ger.gmane.org>
X-Mailman-Original-References: <0589521b-648a-47c1-8afa-6f746e0fb901@googlegroups.com> <lf57fd341sr.fsf@ling.helsinki.fi>
Xref: csiph.com comp.lang.python:110986

On 2016-07-03 08:29, Jussi Piitulainen wrote:
> (Hm. Python seems to understand that the character occurs in what is
> intended to be an identifier. Perhaps that's a default error message.)

I suspect that "identifier" is the final catch-all token in the lexer. Comments 
and strings are clearly delimited. Keywords, operators, and [{(braces)}] are all 
explicitly whitelisted from finite lists. Well, I guess it could have been 
intended by the user to be a numerical literal, but I suspect that's attempted 
before identifier.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco