Path: csiph.com!xmission!news.snarked.org!border2.nntp.dca1.giganews.com!nntp.giganews.com!news.iecc.com!.POSTED.news.iecc.com!nerds-end From: Martin Ward Newsgroups: comp.compilers Subject: Re: Applesoft tokenization phases? Date: Sat, 21 Mar 2020 19:42:17 +0000 Organization: Compilers Central Lines: 32 Sender: news@iecc.com Approved: comp.compilers@iecc.com Message-ID: <20-03-024@comp.compilers> References: <20-03-013@comp.compilers> <20-03-016@comp.compilers> <20-03-017@comp.compilers> <20-03-022@comp.compilers> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Injection-Info: gal.iecc.com; posting-host="news.iecc.com:2001:470:1f07:1126:0:676f:7373:6970"; logging-data="66018"; mail-complaints-to="abuse@iecc.com" Keywords: Basic, lex, history Posted-Date: 21 Mar 2020 16:50:24 EDT X-submission-address: compilers@iecc.com X-moderator-address: compilers-request@iecc.com X-FAQ-and-archives: http://compilers.iecc.com Xref: csiph.com comp.compilers:2495 On 17/03/20 22:14, Ev. Drikos wrote: > Another vague point or simply a point where I'm not really sure that I > translate properly the manual are the reserved keywords before a certain > delimiter. Likely an Applesoft parser must reject this valid UK101 code: > > 10 X=SHIMEM: > 20 END For those less familiar with UK101 BASIC: this code is only valid because "HIMEM" is not a keyword in UK101 BASIC. UK101 does not allow keywords as part of a variable. > There is a reason, BASIC compilers that remove spaces often restrict > variables (like the original Basic version did) to a letter optionally > followed by a digit (and a dollar sign to indicate strings). It makes > the lexing much simpler, even without spaces. Lexing is not the issue: the real reason for the restriction is to make the symbol table much simpler. Each variable in the table takes up a fixed amount of space. For the UK101, only the first two characters of a variable are significant, so only two characters are needed in the symbol table. "Lexing" a statement just consists of replacing all keywords outside strings by special characters. When the interpreted parses it, each statement either starts with a keyword or is an assignment and starts with a variable. -- Martin Dr Martin Ward | Email: martin@gkc.org.uk | http://www.gkc.org.uk G.K.Chesterton site: http://www.gkc.org.uk/gkc | Erdos number: 4