Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed1a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Terry Reedy <tjreedy@udel.edu>
Subject: Re: I need an idea for practise!
Date: Thu, 17 Jul 2014 17:46:51 -0400
References: <6239bcaa-828f-499b-936d-69d022bb94ac@googlegroups.com> <467108ec-19e7-4089-8d5f-53a80244adaf@googlegroups.com> <CAPTjJmr8hmcNpfuC8mgyi=3abuLEhkB0TzOLM6sDArunJWyXMQ@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0
In-Reply-To: <CAPTjJmr8hmcNpfuC8mgyi=3abuLEhkB0TzOLM6sDArunJWyXMQ@mail.gmail.com>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.11964.1405633653.18130.python-list@python.org>
Lines: 47
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:74693

On 7/17/2014 1:20 PM, Chris Angelico wrote:

> By the way, one specific point about RR's advice: A colorizer should
> *not* be written using regexps. It'd make for an absolute nightmare of
> impossible-to-debug regexp strings, plus there are fundamental
> limitations on what you can accomplish with them. You need to use a
> lexer - a lexical analyzer. Basically, to correctly colorize code, you
> need to have something equivalent to the first part of the language
> interpreter, but with a lot more tolerance for errors. That's a pretty
> big thing to write as regexps.

It depends on how deeply one wants to colorize. idlelib.ColorDelegator 
colors comments, strings, keywords, builtin names, and names following 
'def' and 'class' with the following regexes.

def any(name, alternates):
     "Return a named group pattern matching list of alternates."
     return "(?P<%s>" % name + "|".join(alternates) + ")"

def make_pat():
     kw = r"\b" + any("KEYWORD", keyword.kwlist) + r"\b"
     builtinlist = [str(name) for name in dir(builtins)
                                         if not name.startswith('_') and \
                                         name not in keyword.kwlist]
     # self.file = open("file") :
     # 1st 'file' colorized normal, 2nd as builtin, 3rd as string
     builtin = r"([^.'\"\\#]\b|^)" + any("BUILTIN", builtinlist) + r"\b"
     comment = any("COMMENT", [r"#[^\n]*"])
     stringprefix = r"(\br|u|ur|R|U|UR|Ur|uR|b|B|br|Br|bR|BR|rb|rB|Rb|RB)?"
     sqstring = stringprefix + r"'[^'\\\n]*(\\.[^'\\\n]*)*'?"
     dqstring = stringprefix + r'"[^"\\\n]*(\\.[^"\\\n]*)*"?'
     sq3string = stringprefix + r"'''[^'\\]*((\\.|'(?!''))[^'\\]*)*(''')?"
     dq3string = stringprefix + r'"""[^"\\]*((\\.|"(?!""))[^"\\]*)*(""")?'
     string = any("STRING", [sq3string, dq3string, sqstring, dqstring])
     return kw + "|" + builtin + "|" + comment + "|" + string +\
            "|" + any("SYNC", [r"\n"])

prog = re.compile(make_pat(), re.S)
idrog = re.compile(r"\s+(\w+)", re.S)
asprog = re.compile(r".*?\b(as)\b")

I am not sure if the separate definition for as is still needed, or is a 
holdover from when 'as' was not a keyword except in certain contexts.

-- 
Terry Jan Reedy