Re: What stage should entities be resolved?

Path	csiph.com!weretis.net!feeder6.news.weretis.net!news.misty.com!news.iecc.com!.POSTED.news.iecc.com!nerds-end
From	Hans-Peter Diettrich <DrDiettrich1@netscape.net>
Newsgroups	comp.compilers
Subject	Re: What stage should entities be resolved?
Date	Mon, 14 Mar 2022 19:43:22 +0100
Organization	Compilers Central
Lines	28
Sender	news@iecc.com
Approved	comp.compilers@iecc.com
Message-ID	<22-03-031@comp.compilers> (permalink)
References	<22-03-019@comp.compilers> <22-03-025@comp.compilers> <22-03-028@comp.compilers>
Mime-Version	1.0
Content-Type	text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding	8bit
Injection-Info	gal.iecc.com; posting-host="news.iecc.com:2001:470:1f07:1126:0:676f:7373:6970"; logging-data="56396"; mail-complaints-to="abuse@iecc.com"
Keywords	parse, design
Posted-Date	14 Mar 2022 14:50:21 EDT
X-submission-address	compilers@iecc.com
X-moderator-address	compilers-request@iecc.com
X-FAQ-and-archives	http://compilers.iecc.com
Xref	csiph.com comp.compilers:2933

Show key headers only | View raw

On 3/12/22 1:11 PM, Christopher F Clark wrote:
> Contrary to what might assume from my previous posting on this topic.
> I agree with Dodi.
>
> Sometimes, the right answer is another phase.  To keep your lexer
> simple, it can be useful to have a separate phase that deals with
> "character" issues, whether that is transforming UTF-8 extensions into
> unique code points (or actual characters representing glyphs possibly
> accented, i.e. resolving the combining code points into canonical
> versions) or taking sequences like &amp; or \n or whatever into single
> tokens (or characters).  That *can* make the whole process simpler and
> faster.

I consider these "phases" as "filters". In my C parser I also had a
number of filter levels that handle the various aspects in detail of the
preprocessor macro substitution and conditional compilation. The parser
calls the top level filter to return the next C token, which in turn
calls lower level filters until all levels returned enough information
about the next token to parse.

A sloppy interpretation by Microsoft of the preprocessor as a
self-contained stage revealed that the newer C standards disallow a
stand-alone C preprocessor. Such a separate preprocessor could
synthesize tokens like "//" that never occured in a strict (embedded) C
standard implementation. Even if this was not stated explicitly in the
standard it turned out as a side effect of the lexer implementation.

DoDi

Back to comp.compilers | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

What stage should entities be resolved? Lexical analysis stage? Syntax analysis stage? Semantic analysis stage? Roger L Costello <costello@mitre.org> - 2022-03-09 17:22 +0000
  Re: What stage should entities be resolved? Lexical analysis stage? Syntax analysis stage? Semantic analysis stage? Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2022-03-10 09:48 +0100
    Re: What stage should entities be resolved? Christopher F Clark <christopher.f.clark@compiler-resources.com> - 2022-03-12 14:11 +0200
      Re: What stage should entities be resolved? Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2022-03-14 19:43 +0100
    Re: What stage should entities be resolved? Roger L Costello <costello@mitre.org> - 2022-03-15 11:49 +0000
      Re: What stage should entities be resolved? Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2022-03-18 00:31 +0100
      Re: What stage should entities be resolved? gah4 <gah4@u.washington.edu> - 2022-03-17 17:06 -0700
      Re: What stage should entities be resolved? Kaz Kylheku <480-992-1380@kylheku.com> - 2022-03-18 17:50 +0000
        Re: What stage should entities be resolved? gah4 <gah4@u.washington.edu> - 2022-03-18 14:08 -0700
        Re: What stage should entities be resolved? Martin Ward <martin@gkc.org.uk> - 2022-03-19 18:17 +0000
      Re: What stage should entities be resolved? "matt.ti...@gmail.com" <matt.timmermans@gmail.com> - 2022-03-20 07:32 -0700
  RE: What stage should entities be resolved? Christopher F Clark <christopher.f.clark@compiler-resources.com> - 2022-03-10 12:54 +0200
  Re: What stage should entities be resolved? Lexical analysis stage? Syntax analysis stage? Semantic analysis stage? matt.timmermans@gmail.com - 2022-03-12 05:12 -0800

csiph-web