Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.compilers > #3069

The dragon book says separating lexical analysis and parsing is beneficial, so why doesn't ANTLR separate them?

Show all headers | View raw

Since, Terence borrowed that idea from our version of Yacc++, I feel
qualified to answer.  (And we in turn borrowed plenty of ideas from
ANTLR, so it's all fair.)

First, they aren't really merged.  The Scannerless parser people often
merge the idea, but ANTLR still has them separate and generates a
lexer and a parser as two separate entities and there are slight
differences between them (as there should be).  And, you can actually
do them as separate files (and compile them separately) if you want.

All that happens is that you have the two parts using roughly the same
notation (and if you don't need the lexer specific features, it is
essentially a subset of the parsing language, and vice-versa).  So,
you learn that notation and you know it for both parts.

Moreover, by combining the two parts in one file, you know that the
parts "go together" and you have less problems with mismatches,
especially not the kind where you update one but then have an "old"
version of the other which doesn't quite match.  It also allows you to
introduce "tokens" (especially "keywords") in the parser.  (Note you
lose this if you compile the parts separately.)

A slightly more advanced version of that allows you to have tokens
that are only matched in certain contexts.  ANTLR has some of this
implemented, so if you have a place in your parser where you want to
match ">" but not worry about ">>", you can use the literal token in
the grammar and it will override the longest match rule, provided it
doesn't introduce a conflict.  (You also lose this feature with
separate compilation.)

So, note that by keeping the implementations separate (they are really
two phases), you have kept item 1.  Your parser never sees whitepace
and comments, unless you want it to.  You can still do 2 with the
implementation.  I don't know whether the ANTLR generated lexer does
so (or not).  Since ANTLR is Unicode based, 3 is not an issue.

All of this is why we merged the two files into one in Yacc++ and
ANLTR started doing the same thing.  We also merged them because our
original target was going to be building a syntax directed editor
version of Emacs and these ideas made sense in that regard.

--
******************************************************************************
Chris Clark                  email: christopher.f.clark@compiler-resources.com
Compiler Resources, Inc.  Web Site: http://world.std.com/~compres
23 Bailey Rd                 voice: (508) 435-5016
Berlin, MA  01503 USA      twitter: @intel_chris
------------------------------------------------------------------------------

Back to comp.compilers | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

The dragon book says separating lexical analysis and parsing is beneficial, so why doesn't ANTLR separate them? Roger L Costello <costello@mitre.org> - 2022-06-09 14:52 +0000
  Re: The dragon book says separating lexical analysis and parsing is beneficial, so why doesn't ANTLR separate them? "Alexei A. Frounze" <alexfrunews@gmail.com> - 2022-06-09 18:07 -0700
  Re: The dragon book says separating lexical analysis and parsing is beneficial, so why doesn't ANTLR separate them? gah4 <gah4@u.washington.edu> - 2022-06-09 23:01 -0700
  Re: The dragon book says separating lexical analysis and parsing is beneficial, so why doesn't ANTLR separate them? Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2022-06-10 12:26 +0200
  The dragon book says separating lexical analysis and parsing is beneficial, so why doesn't ANTLR separate them? Christopher F Clark <christopher.f.clark@compiler-resources.com> - 2022-06-11 23:45 +0300
  Re: The dragon book says separating lexical analysis and parsing is beneficial, so why doesn't ANTLR separate them? George Neuner <gneuner2@comcast.net> - 2022-06-11 18:15 -0400
    Re: The dragon book says separating lexical analysis and parsing is beneficial, so why doesn't ANTLR separate them? anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2022-06-12 14:10 +0000

csiph-web