Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.compilers > #2325

Regular expressions in lexing and parsing

Show all headers | View raw

 >[Orignally here
https://commandcenter.blogspot.com/2011/08/regular-expressions-in-lexing-and.html

 >It took me a minute to figure out what he was saying, since the patterns in a lexer are regular
 >expressions. I believe the point is not to use general purpose regex libraries but rather to
 >use something like flex or re2c which will take a set of expressions and actions and
 >precompile them. -John]

I have my doubts due to:

 >Consider finding
 >alphanumeric identifiers. It's not too hard to write the regexp (something
 >like "[a-ZA-Z_][a-ZA-Z_0-9]*"), but really not much harder to write as a
 >simple loop. The performance of the loop, though, will be much higher and will
 >involve much less code under the covers.

and

 >And when we want to adjust our lexer to admit other character
 >types, such as Unicode identifiers, and handle normalization, and so on, the
 >hand-written loop can cope easily but the regexp approach will break down.

It doesn't help that flex *still* doesn't support Unicode, but according
to http://re2c.org/manual/features/encodings/encodings.html re2c does.

My view is that just as regex libraries have been embraced by most
languages (even C++ has std::regex now), the same should happen for
lexer and parser generators. This is the approach I have taken with
lexertl and parsertl. Hana Dusíková is doing interesting work in the
compile time regex space with
https://github.com/hanickadot/compile-time-regular-expressions .
Apparently she is working on a DFA version this year which will be
presented at cppnow - http://cppnow.org/history/2019/schedule/ . I'm
hoping that this new version can be used as a lexer generator and then I
can switch to it for a lot of my lexing needs.

Regards,

Ben
[I agree that once you have some way to embed precompiled DFAs in your
code, it doesn't much matter whether the rest is  pattern-action like
in flex or something else. -John]

Back to comp.compilers | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Regular expressions in lexing and parsing Ed Davis <ed_davis2@yahoo.com.dmarc.email> - 2019-05-17 13:18 +0000
  Regular expressions in lexing and parsing Ben Hanson <jamin.hanson@googlemail.com> - 2019-05-18 13:24 +0100
  Re: Regular expressions in lexing and parsing Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2019-05-21 07:14 +0200
    Re: Regular expressions in lexing and parsing Fortran steve kargl <sgk@troutmask.apl.washington.edu> - 2019-05-22 04:12 +0000
    Re: Regular expressions in lexing and parsing "Ev. Drikos" <drikosev@gmail.com> - 2019-05-23 18:42 +0300

csiph-web