Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.compilers > #1913

Re: re2c-1.0 released!

Show key headers only | View raw

On 2017-09-02, Anton Ertl <anton@mips.complang.tuwien.ac.at> wrote:
> Kaz Kylheku <398-816-0742@kylheku.com> writes:
>>Briefly, why would you do some hacky regex thing in lex with \1, \2,
>>\3, when in the level immediately above yylex() you have proper phrase
>>recognition, with $1, $2, $3.
>
> These don't do the same thing.  Taking your example from the other
> posting, do you want to recognize
>
> 2017
> -08
> /* bla bla */ -
> 28
>
> as date?  If so, do it at the parser level, if not, the scanner.

We face this choice only because we put a hack into the lexer:
when it scans whitespace token or comment token, it throws it away,
so the parser doesn't see it.

That hack is very easy to implement and considerably simplifies the
grammar.

We can very easily implement another hack to allow us to recognize
2017-08-28 as three tokens without comments or whitespace.

In fact we can do it all in standard lex.

We can set up a custom YYINPUT which allows the entire yytext to be
pushed back into the stream.  The rest is done with orchestration of one
or more start states.

The steps to recognizing a date would then be:

1. First NNNN-NN-NN is scanned as a token.
2. push_back_string(yytext) is called
3. BEGIN(DATE) is invoked
4. The rule doesn't return so the lexer re-scans the pushed input
   in the DATE state.
5. In the DATE state, an integer token is recognized; everything
   else is an error. Dash tokens can be recognized and returned,
   or consumed in the lexer.
6. Successful recognition of an integer token in the DATE state
   returns not an INTEGER to the parser but a DINTEGER.
7. The parser's phrase structure for matching dates refers to DINTEGER
   nonterminals and not INTEGER.

Pseudo-code, including a mechanism for returning to the INITIAL state
without parser involvement:

%x DATE

%%

<INITIAL,DATE>[0-9]+ {
  yylval.value = str_to_int(yytext);  /* our function */
  return (YYSTATE == DATE) ? DINTEGER : INTEGER;
}

<INITIAL,DATE>- {
  return '-';
}

[0-9]+-[0-9]+-[0-9]+ {
  unput('!');           /* end signal */
  unput_string(yytext); /* our function; works with our YYINPUT */
  BEGIN(DATE);
}

<DATE>! {               /* our end signal: include other states here */
  BEGIN(INITIAL);
}

<DATE>. {
  /* internal error: how did we get here? */
}

Back to comp.compilers | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

re2c-1.0 released! Ulya Trofimovich <skvadrik@gmail.com> - 2017-08-27 18:25 +0100
  Re: re2c-1.0 released! Kaz Kylheku <398-816-0742@kylheku.com> - 2017-09-02 11:18 -0400
    Re: re2c-1.0 released! anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2017-09-02 17:02 +0000
      Re: re2c-1.0 released! Kaz Kylheku <398-816-0742@kylheku.com> - 2017-09-03 15:14 +0000
      Re: re2c-1.0 released! Ulya Trofimovich <skvadrik@gmail.com> - 2017-09-03 17:58 +0100
      Re: re2c-1.0 released! Ulya Trofimovich <skvadrik@gmail.com> - 2017-09-03 18:31 +0100
        Re: re2c-1.0 released! Ben Hanson <jamin.hanson@googlemail.com> - 2017-09-04 05:19 -0700
          Re: re2c-1.0 released! Ulya Trofimovich <skvadrik@gmail.com> - 2017-09-08 23:40 +0100
  Re: re2c-1.0 released! Kaz Kylheku <398-816-0742@kylheku.com> - 2017-09-02 11:18 -0400
  Re: re2c-1.0 released! George Neuner <gneuner2@comcast.net> - 2017-09-02 14:11 -0400
    Re: re2c-1.0 released! Ulya Trofimovich <skvadrik@gmail.com> - 2017-09-03 09:00 +0100
    Re: re2c-1.0 released! Ben Hanson <jamin.hanson@googlemail.com> - 2017-09-03 03:13 -0700
  Re: re2c-1.0 released! Ben Hanson <jamin.hanson@googlemail.com> - 2017-09-03 03:16 -0700

csiph-web