Re: Languages with optional spaces

From	awanderin <awanderin@gmail.com>
Newsgroups	comp.compilers
Subject	Re: Languages with optional spaces
Date	2020-02-26 10:03 -0700
Organization	A noiseless patient Spider
Message-ID	<20-02-022@comp.compilers> (permalink)
References	<20-02-015@comp.compilers> <20-02-016@comp.compilers> <20-02-019@comp.compilers>

Show all headers | View raw

Maury Markowitz <maury.markowitz@gmail.com> writes:

> Unless I'm mistaken, Integer BASIC is a "deep tokenizer", so is it not
> the case the spaces exist only at entry time? IE, if one types
> 'PRINT"HELLO"' that will actually be LISTed as 'PRINT "HELLO"?
> [That is my recollection.  Storing tokens takes less space than
> storing the input text and on those tiny little
> micros, every byte counted. -John]

In the Apple II Microsoft BASIC, yes, the tokenizer removed spaces
except within strings.  On the Commodore machines, their version of
Microsoft BASIC did not purge the spaces.  This allowed the programmer
to "indent" the code (after the line number).  The tokens, like PRINT,
FOR, INPUT, etc, are stored as one byte in all the Microsoft BASICs.
The Apple II Integer BASIC also tokenizes to one byte and removes
spaces, but it did more syntax checking at entry time, so it was more a
lexer + parser than a lexer like Microsoft BASIC.

--
--
Jerry    awanderin at gmail dot com

Back to comp.compilers | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Languages with optional spaces Maury Markowitz <maury.markowitz@gmail.com> - 2020-02-19 07:35 -0800
  Re: Languages with optional spaces Jerry <awanderin@gmail.com> - 2020-02-20 23:38 -0700
    Re: Languages with optional spaces Maury Markowitz <maury.markowitz@gmail.com> - 2020-02-25 06:13 -0800
      Re: Languages with optional spaces awanderin <awanderin@gmail.com> - 2020-02-26 10:03 -0700
    Re: Languages with optional spaces "Ev. Drikos" <drikosev@gmail.com> - 2020-03-12 17:45 +0200
  Re: Languages with optional spaces "Ev. Drikos" <drikosev@gmail.com> - 2020-02-23 12:33 +0200
    Re: Languages with optional spaces Martin Ward <martin@gkc.org.uk> - 2020-02-25 17:00 +0000
      Re: Languages with optional spaces "Ev. Drikos" <drikosev@gmail.com> - 2020-02-28 13:34 +0200
    Re: Languages with optional spaces Christopher F Clark <christopher.f.clark@compiler-resources.com> - 2020-02-29 11:48 +0200
      Re: Languages with optional spaces "Ev. Drikos" <drikosev@gmail.com> - 2020-02-29 21:38 +0200
        Re: Languages with optional spaces Christopher F Clark <christopher.f.clark@compiler-resources.com> - 2020-03-01 10:07 +0200
        Re: Languages with optional spaces "Ev. Drikos" <drikosev@gmail.com> - 2020-03-01 19:41 +0200
          Re: Languages with optional spaces Christopher F Clark <christopher.f.clark@compiler-resources.com> - 2020-03-02 08:33 +0200
            Re: Languages with optional spaces "Ev. Drikos" <drikosev@gmail.com> - 2020-03-02 20:04 +0200
      Re: Languages with optional spaces Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2020-03-01 00:28 +0100
  Re: Languages with optional spaces Maury Markowitz <maury.markowitz@gmail.com> - 2020-02-25 06:11 -0800
  Re: Languages with optional spaces Kaz Kylheku <493-878-3164@kylheku.com> - 2020-02-26 08:06 +0000
  Re: Languages with optional spaces and tools Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2020-02-28 20:16 +0100
  Re: Languages with optional spaces gah4@u.washington.edu - 2020-03-02 21:12 -0800
  Re: Languages with optional spaces Gene <gene.ressler@gmail.com> - 2020-04-14 10:08 -0700
  Re: Languages with optional spaces mertesthomas@gmail.com - 2020-04-19 04:04 -0700
  Re: Languages with optional spaces aston.goldsmith@gmail.com - 2020-05-05 13:05 -0700

csiph-web