Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.compilers > #3004

Re: Please provide a learning path for mastering lexical analysis languages

Show all headers | View raw

/* Token Rules */

	<eof>			-> \z

	<constant>     	-> literal
					-> integer
					-> decimal
					-> real

	<identifier>   		-> letter (letter|digit)*

	integer        		-> digit+

	real	         		-> integer exp
					-> decimal exp

	decimal        		-> digit+ '.'
					->        '.' digit+
					-> digit+ '.' digit+

	exp	         		-> 'e' digit+
					-> 'E' digit+
					-> 'e' '-' digit+
					-> 'E' '-' digit+
					-> 'e' '+' digit+
					-> 'E' '+' digit+

	literal        		-> ''' lchar '''

	lchar          		-> lany
					-> '\' '\'
					-> '\' '''
					-> '\' '"'
					-> '\' 'n'
					-> '\' 't'
					-> '\' 'a'
					-> '\' 'b'
					-> '\' 'f'
					-> '\' 'r'
					-> '\' 'v'
					-> '\' '0'

	<string>       		-> '"' schar* '"'

	schar          		-> sany
					-> '\' '\'
					-> '\' '''
					-> '\' '"'
					-> '\' 'n'
					-> '\' 't'
					-> '\' 'a'
					-> '\' 'b'
					-> '\' 'f'
					-> '\' 'r'
					-> '\' 'v'
					-> '\' '0'

	{whitespace}       	-> whitechar+

	{commentline}  	-> '/' '/' neol*

	{commentblock} 	-> '/' '*' na* '*'+ (nans na* '*'+)* '/'

/* Character Sets */

	any				= 0..255 - \z
	lany           		= any - ''' - '\' - \n
	sany	         	= any - '"' - '\' - \n

	letter	      		= 'a'..'z' | 'A'..'Z' | '_'
	digit	         	= '0'..'9'

	whitechar     		= \t | \n | \r | \f | \v | ' '

	na             		= any - '*' 			// not asterisk
	nans           		= any - '*' - '/'	                // not asterisk not slash
	neol           		= any - \n 			// not end of line

	\t				=  9					// tab
	\n				= 10				// newline
	\v				= 11				// vertical feed?
	\f				= 12				// form feed
	\r				= 13				// return
	\z				= 26				// end of file
	\b				= 32				// blank/space

/* End */

The above lexical rules define C-language symbols.
It's just a lexical grammar, not too hard to figure out.
This is input to the DFA lexer generator, which is provided
with the LRSTAR parser generator on SourceForge.net.

DFA creates lexers that run 80% faster than "flex" lexers
and are about the same size.

If you need more language power to define a lexer ...
that's what parser are for.

BTW, LRSTAR creates parsers in C++ than were running
140 times faster than those created by ANTLR, using the
C++ target, the last time I did a comparison, 2 years ago.

Back to comp.compilers | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Please provide a learning path for mastering lexical analysis languages Roger L Costello <costello@mitre.org> - 2022-05-06 11:59 +0000
  Re: Please provide a learning path for mastering lexical analysis languages George Neuner <gneuner2@comcast.net> - 2022-05-08 12:52 -0400
    Re: Please provide a learning path for mastering lexical analysis languages Paul B Mann <parser.generator.guy@gmail.com> - 2022-05-08 22:27 -0700
    Re: Please provide a learning path for mastering lexical analysis languages Christopher F Clark <christopher.f.clark@compiler-resources.com> - 2022-05-22 12:12 +0300
  Re: Please provide a learning path for mastering lexical analysis languages luser droog <luser.droog@gmail.com> - 2022-05-08 18:08 -0700
  Re: Please provide a learning path for mastering lexical analysis languages gah4 <gah4@u.washington.edu> - 2022-05-13 13:42 -0700

csiph-web