Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.compilers > #2338

Re: Regular expressions in lexing and parsing

Show key headers only | View raw

On Tue, Jun 18, 2019 at 12:23 PM John wrote:
> [I agree with the sentiment to use formal grammars for both lex and
> parse, but it always has seemed to me that running them separately as
> coroutines makes it easier to deal with comments and whitespace
> without having to hang them on every token definition.  I realize that
> the more context sensitive the language, the more it is likely to make
> sense to combine them. -John]

The Meta-S system has a way to do that automatically. The "whitespace
rule" (__ws is the reserved name for it) also has two alias sequences
that make it simple to specify optional versus required whitespace (#@
and ## respectively):

some_rule ## some_other_rule // whitespace is not optional

some_rule #? "(" #? some_other_rule #? ")" // whitespace in all
positions is optional

Moreover, the definition of the __ws can be:

__ws ::= #notree #standard_whitespace; // defined as standard C type
whitespace, and don't stick it on the AST when recognized in the input

I have found that, while sometimes verbose, this explicit handling
makes the dynamic nature of the grammars far easier to implement at
the back-end. (At the cost of the human grammar author having to think
about whitespace more often than some would prefer.)

If I recall correctly, it was Hans-Peter Dietrich (the OP) who
convinced me a number of years back to include the ability to specify
certain tokens that did not need explicit whitespace around them in
the productions that use them. (The grammar compiler itself then takes
responsibility for inserting the appropriate whitespace operators
behind the scenes during compilation when those tokens are used.)

In the long run, this hybrid approach has offered the most
versatility, without being too cumbersome. (There are bigger dragons
to fight with adaptive systems that can modify their rules in
mid-parse.....)

--
Quinn Jackson CSci MIScT SMACM SMIEEE FRSA

LinkedIn:     http://ca.linkedin.com/in/quinnjackson/
ResearchGate: http://researchgate.net/profile/Quinn_Jackson/

Back to comp.compilers | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Re: Regular expressions in lexing and parsing Christopher F Clark <christopher.f.clark@compiler-resources.com> - 2019-06-17 08:37 -0400
  Re: Regular expressions in lexing and parsing Quinn Jackson <quinn.jackson@ieee.org> - 2019-06-18 11:56 -0700
    Re: Regular expressions in lexing and parsing Quinn Jackson <quinn.jackson@ieee.org> - 2019-06-18 12:57 -0700
    Re: Regular expressions in lexing and parsing Kaz Kylheku <847-115-0292@kylheku.com> - 2019-06-18 20:01 +0000
  Re: Regular expressions in lexing and parsing Christopher F Clark <christopher.f.clark@compiler-resources.com> - 2019-06-18 18:43 -0400

csiph-web