Path: csiph.com!xmission!news.snarked.org!border2.nntp.dca1.giganews.com!nntp.giganews.com!news.iecc.com!.POSTED.news.iecc.com!nerds-end
From: "Ev. Drikos" <drikosev@gmail.com>
Newsgroups: comp.compilers
Subject: Re: Languages with optional spaces
Date: Fri, 28 Feb 2020 13:34:40 +0200
Organization: Aioe.org NNTP Server
Lines: 40
Sender: news@iecc.com
Approved: comp.compilers@iecc.com
Message-ID: <20-02-030@comp.compilers>
References: <20-02-015@comp.compilers> <20-02-017@comp.compilers> <20-02-020@comp.compilers>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: gal.iecc.com; posting-host="news.iecc.com:2001:470:1f07:1126:0:676f:7373:6970"; logging-data="99116"; mail-complaints-to="abuse@iecc.com"
Keywords: lex, Basic
Posted-Date: 28 Feb 2020 12:28:09 EST
X-submission-address: compilers@iecc.com
X-moderator-address: compilers-request@iecc.com
X-FAQ-and-archives: http://compilers.iecc.com
Content-Language: en-US
Xref: csiph.com comp.compilers:2467

On 25/02/2020 19:00, Martin Ward wrote:
> ...So any statement either starts
> with a keyword (a control character), which determines
> the statement type, or it is an assignment with the LET
> keyword omitted...


A scanner generator that supports ie "difference" operators can build a
DFA that splits ie the text FORI to FOR & I without space overhead, if
the reserved keywords are also tokens of the lexer.

The lexical rules below result ie to a DFA with 10 states. Even if I
erase the difference operator "-=" till the end of the document, the
DFA built still has 10 states but the text FORI is returned as an "id".

So, the rules below ensure that an "id" doesn't start with FOR or PRINT,
not sure though if this is what the OP really wanted once he said that
in theory FORI could be a variable (perhaps it can't be just an LHS?).


Ev. Drikos

-----------------------------------------------------------------------
token ::=
            FOR
      |     PRINT
      |     id

FOR ::=
            F O R

PRINT ::=
            P R I N T

id ::=
            {A..Z [{$|A .. Z|0 .. 9}...]} -= {key[{$|A .. Z|0..9}...]}

key ::=
            FOR
      |     PRINT