Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.compilers > #2931

Re: What stage should entities be resolved?

Show all headers | View raw

Contrary to what might assume from my previous posting on this topic.
I agree with Dodi.

Sometimes, the right answer is another phase.  To keep your lexer
simple, it can be useful to have a separate phase that deals with
"character" issues, whether that is transforming UTF-8 extensions into
unique code points (or actual characters representing glyphs possibly
accented, i.e. resolving the combining code points into canonical
versions) or taking sequences like & or \n or whatever into single
tokens (or characters).  That *can* make the whole process simpler and
faster.

For example, years ago when working on a C compiler for Honeywell when
the first ANSI standard was still new, the standard had 8 stages (if I
recall correctly) that described the lexing process.  We decided that
the best way to assure faithfulness to the standard was to implement
the 8 stages exactly as specified, at least in the first version.
That way we had a reliable model of the desired behavior that we could
track back to the standard.  Moreover, by having them as separate
pieces of code, it was easy to turn them off (e.g. trigraphs in C were
an ANSI invention and some C programs used ??? not as a trigraph but
as a way of emphasis).   Similarly, some pre-ANSI C dialects supported
nested comments and you might want to change that phase.

While you do want each phase to generally build larger and larger
structures.  I.e. you don't want your parser very often dealing with
strings as individual characters.  The exact number of phases or
content of each phase can vary slightly.  One size rarely fits all.

--
******************************************************************************
Chris Clark                  email: christopher.f.clark@compiler-resources.com
Compiler Resources, Inc.  Web Site: http://world.std.com/~compres
23 Bailey Rd                 voice: (508) 435-5016
Berlin, MA  01503 USA      twitter: @intel_chris
------------------------------------------------------------------------------

Back to comp.compilers | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

What stage should entities be resolved? Lexical analysis stage? Syntax analysis stage? Semantic analysis stage? Roger L Costello <costello@mitre.org> - 2022-03-09 17:22 +0000
  Re: What stage should entities be resolved? Lexical analysis stage? Syntax analysis stage? Semantic analysis stage? Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2022-03-10 09:48 +0100
    Re: What stage should entities be resolved? Christopher F Clark <christopher.f.clark@compiler-resources.com> - 2022-03-12 14:11 +0200
      Re: What stage should entities be resolved? Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2022-03-14 19:43 +0100
    Re: What stage should entities be resolved? Roger L Costello <costello@mitre.org> - 2022-03-15 11:49 +0000
      Re: What stage should entities be resolved? Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2022-03-18 00:31 +0100
      Re: What stage should entities be resolved? gah4 <gah4@u.washington.edu> - 2022-03-17 17:06 -0700
      Re: What stage should entities be resolved? Kaz Kylheku <480-992-1380@kylheku.com> - 2022-03-18 17:50 +0000
        Re: What stage should entities be resolved? gah4 <gah4@u.washington.edu> - 2022-03-18 14:08 -0700
        Re: What stage should entities be resolved? Martin Ward <martin@gkc.org.uk> - 2022-03-19 18:17 +0000
      Re: What stage should entities be resolved? "matt.ti...@gmail.com" <matt.timmermans@gmail.com> - 2022-03-20 07:32 -0700
  RE: What stage should entities be resolved? Christopher F Clark <christopher.f.clark@compiler-resources.com> - 2022-03-10 12:54 +0200
  Re: What stage should entities be resolved? Lexical analysis stage? Syntax analysis stage? Semantic analysis stage? matt.timmermans@gmail.com - 2022-03-12 05:12 -0800

csiph-web