[PC] semi-lazy tokenizer

Path	csiph.com!eternal-september.org!feeder.eternal-september.org!reader01.eternal-september.org!.POSTED!not-for-mail
From	M Joshua Ryan <luser.droog@gmail.com>
Newsgroups	comp.lang.c
Subject	[PC] semi-lazy tokenizer
Date	Thu, 11 Apr 2019 16:56:40 -0500
Organization	A noiseless patient Spider
Lines	43
Message-ID	<vriubm1ctdd3.fsf@gmail.com> (permalink)
Mime-Version	1.0
Content-Type	text/plain
Injection-Info	reader02.eternal-september.org; posting-host="dd859abbdece10ed1652a4647c39d22a"; logging-data="1079"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX18695cn6q8Hu/NgcXGbJ9i8T4g9k1bXvT8="
User-Agent	Gnus/5.13 (Gnus v5.13) Emacs/25.3 (cygwin)
Cancel-Lock	sha1:Om/6PLn/x+PgGga3k5YSzkt/kH0= sha1:Tvodk2fK7KHCisHlGaqdD/t6FQU=
Xref	csiph.com comp.lang.c:143041

Show key headers only | View raw

As discovered by Ben in a previous thread, I'm not as lazy as
I could be (but expending great effort to become so). But even
with half-lazy the code has met my prediction that it becomes
relatively painless to abstract over lists of tokens with the
same tools that work on lists of characters. So, my recent 
exercise shows a tokenizer built using parser combinators
which presents a *token stream* interface which could then
be used by a further layer of parsers.


list token_input( void *s ){
  if(  !s  ) return NULL;
  static parser p;
  if(  !p  ){
    parser space   = many( anyof( " " ) );
    parser digits  = some( is_digit() );
    parser letters = some( is_alpha() );
    p = seq( space, plus( digits, letters ) );
  }
  list r = x( parse( p, s ) );
  return  cons( x( r ), Suspension( xs( r ), token_input ) );
}

void test_tokens(){
  object tokens = token_input( string_input( "123 blah bluh bleh" ) );
  print( take( 2, tokens ) ), puts("");
  print( drop( 2, tokens ) ), puts("");
  print( take( 2, drop( 2, tokens ) ) ), puts("");
}


which has this output:
make -k
cc -std=c99    -c -o pc7.o pc7.c
cc -std=c99  -o pc7_test pc7_test.c pc7.o 
./pc7_test
((() (49 (50 (51 () ) ) ) ) (((32 () ) (98 (108 (97 (104 () ) ) ) ) ) () ) ) 
(((32 () ) (98 (108 (117 (104 () ) ) ) ) ) SUSPENSION ) 
(((32 () ) (98 (108 (117 (104 () ) ) ) ) ) (((32 () ) (98 (108 (101 (104 () ) ) ) ) ) () ) ) 

Which is the desired result. First two tokens extracted on the first
line. Where each token is, so far, just a list of its characters
prepended with any whitespace characters.

Back to comp.lang.c | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

[PC] semi-lazy tokenizer M Joshua Ryan <luser.droog@gmail.com> - 2019-04-11 16:56 -0500
  Re: [PC] semi-lazy tokenizer Chad <cdalten@gmail.com> - 2019-04-11 15:18 -0700
  Re: [PC] semi-lazy tokenizer M Joshua Ryan <luser.droog@gmail.com> - 2019-04-13 00:07 -0500
    Re: [PC] semi-lazy tokenizer fir <profesor.fir@gmail.com> - 2019-04-14 12:21 -0700
      Re: [PC] semi-lazy tokenizer fir <profesor.fir@gmail.com> - 2019-04-14 12:47 -0700
      Re: [PC] semi-lazy tokenizer luser droog <luser.droog@gmail.com> - 2019-04-14 23:05 -0700
        Re: [PC] semi-lazy tokenizer fir <profesor.fir@gmail.com> - 2019-04-15 00:26 -0700
          Re: [PC] semi-lazy tokenizer fir <profesor.fir@gmail.com> - 2019-04-15 00:51 -0700
            Re: [PC] semi-lazy tokenizer fir <profesor.fir@gmail.com> - 2019-04-15 07:06 -0700
              Re: [PC] semi-lazy tokenizer fir <profesor.fir@gmail.com> - 2019-04-15 07:13 -0700

csiph-web