Path: csiph.com!xmission!news.snarked.org!border2.nntp.dca1.giganews.com!nntp.giganews.com!news.iecc.com!.POSTED.news.iecc.com!nerds-end From: awanderin Newsgroups: comp.compilers Subject: Re: Languages with optional spaces Date: Wed, 26 Feb 2020 10:03:11 -0700 Organization: A noiseless patient Spider Lines: 21 Sender: news@iecc.com Approved: comp.compilers@iecc.com Message-ID: <20-02-022@comp.compilers> References: <20-02-015@comp.compilers> <20-02-016@comp.compilers> <20-02-019@comp.compilers> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: gal.iecc.com; posting-host="news.iecc.com:2001:470:1f07:1126:0:676f:7373:6970"; logging-data="4585"; mail-complaints-to="abuse@iecc.com" Keywords: lex, Basic, history Posted-Date: 27 Feb 2020 17:34:05 EST X-submission-address: compilers@iecc.com X-moderator-address: compilers-request@iecc.com X-FAQ-and-archives: http://compilers.iecc.com Xref: csiph.com comp.compilers:2460 Maury Markowitz writes: > Unless I'm mistaken, Integer BASIC is a "deep tokenizer", so is it not > the case the spaces exist only at entry time? IE, if one types > 'PRINT"HELLO"' that will actually be LISTed as 'PRINT "HELLO"? > [That is my recollection. Storing tokens takes less space than > storing the input text and on those tiny little > micros, every byte counted. -John] In the Apple II Microsoft BASIC, yes, the tokenizer removed spaces except within strings. On the Commodore machines, their version of Microsoft BASIC did not purge the spaces. This allowed the programmer to "indent" the code (after the line number). The tokens, like PRINT, FOR, INPUT, etc, are stored as one byte in all the Microsoft BASICs. The Apple II Integer BASIC also tokenizes to one byte and removes spaces, but it did more syntax checking at entry time, so it was more a lexer + parser than a lexer like Microsoft BASIC. -- -- Jerry awanderin at gmail dot com