Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #143066

Re: [PC] semi-lazy tokenizer

Show key headers only | View raw

M Joshua Ryan <luser.droog@gmail.com> writes:

> As discovered by Ben in a previous thread, I'm not as lazy as
> I could be (but expending great effort to become so). But even
> with half-lazy the code has met my prediction that it becomes
> relatively painless to abstract over lists of tokens with the
> same tools that work on lists of characters. So, my recent 
> exercise shows a tokenizer built using parser combinators
> which presents a *token stream* interface which could then
> be used by a further layer of parsers.
>

Some further enhancements make the output nicer and the results
more useful. I've added functions to convert a list of characters
into a string or and integer. And a constructor for "symbols" 
which map to enum values (but also have a printable name string).

Hooking these into the tokenizer looks like this:


enum extended_symbols { t_int = USER_SYMBOL, t_num };

object on_digits( void *v, list o ){ return  cons( Symbol(t_int), to_integer( o ) ); }
object on_letters( void *v, list o ){ return  cons( Symbol(t_num), to_string( o ) ); }
list token_input( void *s ){
  if(  !s  ) return NULL;
  static parser p;
  if(  !p  ){
    parser space   = many( anyof( " " ) );
    parser digits  = using( some( is_digit() ), on_digits  );
    parser letters = using( some( is_alpha() ), on_letters );
    p = seq( space, plus( digits, letters ) );
  }
  list r = x( parse( p, s ) );
  return  cons( x( r ), Suspension( xs( r ), token_input ) );
}


And produces this (nicer) output:

make -k
cc -std=c99    -c -o pc7.o pc7.c
cc -std=c99  -o pc7_test pc7_test.c pc7.o 
./pc7_test
((() (t_int 123 ) ) (((32 () ) (t_num "blah" ) ) () ) ) 
(((32 () ) (t_num "bluh" ) ) SUSPENSION ) 
(((32 () ) (t_num "bluh" ) ) (((32 () ) (t_num "bleh" ) ) () ) ) 


The next level of parsers can operate on these "token objects" 
by defining predicates and using the 'parser sat( predicate pred )' 
parser constructor which is type agnostic as far as the elements
of the input list. 

I suppose the next step is to try a real grammar, maybe an early
version of C...

Back to comp.lang.c | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[PC] semi-lazy tokenizer M Joshua Ryan <luser.droog@gmail.com> - 2019-04-11 16:56 -0500
  Re: [PC] semi-lazy tokenizer Chad <cdalten@gmail.com> - 2019-04-11 15:18 -0700
  Re: [PC] semi-lazy tokenizer M Joshua Ryan <luser.droog@gmail.com> - 2019-04-13 00:07 -0500
    Re: [PC] semi-lazy tokenizer fir <profesor.fir@gmail.com> - 2019-04-14 12:21 -0700
      Re: [PC] semi-lazy tokenizer fir <profesor.fir@gmail.com> - 2019-04-14 12:47 -0700
      Re: [PC] semi-lazy tokenizer luser droog <luser.droog@gmail.com> - 2019-04-14 23:05 -0700
        Re: [PC] semi-lazy tokenizer fir <profesor.fir@gmail.com> - 2019-04-15 00:26 -0700
          Re: [PC] semi-lazy tokenizer fir <profesor.fir@gmail.com> - 2019-04-15 00:51 -0700
            Re: [PC] semi-lazy tokenizer fir <profesor.fir@gmail.com> - 2019-04-15 07:06 -0700
              Re: [PC] semi-lazy tokenizer fir <profesor.fir@gmail.com> - 2019-04-15 07:13 -0700

csiph-web