Path: csiph.com!goblin2!goblin3!goblin.stu.neva.ru!news.misty.com!news.iecc.com!.POSTED.news.iecc.com!nerds-end From: Quinn Jackson Newsgroups: comp.compilers Subject: Re: Regular expressions in lexing and parsing Date: Tue, 18 Jun 2019 12:57:39 -0700 Organization: Compilers Central Lines: 45 Sender: news@iecc.com Approved: comp.compilers@iecc.com Message-ID: <19-06-009@comp.compilers> References: <19-06-005@comp.compilers> <19-06-008@comp.compilers> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Injection-Info: gal.iecc.com; posting-host="news.iecc.com:2001:470:1f07:1126:0:676f:7373:6970"; logging-data="15907"; mail-complaints-to="abuse@iecc.com" Keywords: lex, parse Posted-Date: 18 Jun 2019 16:17:36 EDT X-submission-address: compilers@iecc.com X-moderator-address: compilers-request@iecc.com X-FAQ-and-archives: http://compilers.iecc.com In-Reply-To: <19-06-008@comp.compilers> Xref: csiph.com comp.compilers:2338 On Tue, Jun 18, 2019 at 12:23 PM John wrote: > [I agree with the sentiment to use formal grammars for both lex and > parse, but it always has seemed to me that running them separately as > coroutines makes it easier to deal with comments and whitespace > without having to hang them on every token definition. I realize that > the more context sensitive the language, the more it is likely to make > sense to combine them. -John] The Meta-S system has a way to do that automatically. The "whitespace rule" (__ws is the reserved name for it) also has two alias sequences that make it simple to specify optional versus required whitespace (#@ and ## respectively): some_rule ## some_other_rule // whitespace is not optional some_rule #? "(" #? some_other_rule #? ")" // whitespace in all positions is optional Moreover, the definition of the __ws can be: __ws ::= #notree #standard_whitespace; // defined as standard C type whitespace, and don't stick it on the AST when recognized in the input I have found that, while sometimes verbose, this explicit handling makes the dynamic nature of the grammars far easier to implement at the back-end. (At the cost of the human grammar author having to think about whitespace more often than some would prefer.) If I recall correctly, it was Hans-Peter Dietrich (the OP) who convinced me a number of years back to include the ability to specify certain tokens that did not need explicit whitespace around them in the productions that use them. (The grammar compiler itself then takes responsibility for inserting the appropriate whitespace operators behind the scenes during compilation when those tokens are used.) In the long run, this hybrid approach has offered the most versatility, without being too cumbersome. (There are bigger dragons to fight with adaptive systems that can modify their rules in mid-parse.....) -- Quinn Jackson CSci MIScT SMACM SMIEEE FRSA LinkedIn: http://ca.linkedin.com/in/quinnjackson/ ResearchGate: http://researchgate.net/profile/Quinn_Jackson/