Path: csiph.com!eternal-september.org!feeder.eternal-september.org!reader01.eternal-september.org!.POSTED!not-for-mail
From: M Joshua Ryan <luser.droog@gmail.com>
Newsgroups: comp.lang.c
Subject: [PC] semi-lazy tokenizer
Date: Thu, 11 Apr 2019 16:56:40 -0500
Organization: A noiseless patient Spider
Lines: 43
Message-ID: <vriubm1ctdd3.fsf@gmail.com>
Mime-Version: 1.0
Content-Type: text/plain
Injection-Info: reader02.eternal-september.org; posting-host="dd859abbdece10ed1652a4647c39d22a"; logging-data="1079"; mail-complaints-to="abuse@eternal-september.org";	posting-account="U2FsdGVkX18695cn6q8Hu/NgcXGbJ9i8T4g9k1bXvT8="
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (cygwin)
Cancel-Lock: sha1:Om/6PLn/x+PgGga3k5YSzkt/kH0= sha1:Tvodk2fK7KHCisHlGaqdD/t6FQU=
Xref: csiph.com comp.lang.c:143041

As discovered by Ben in a previous thread, I'm not as lazy as
I could be (but expending great effort to become so). But even
with half-lazy the code has met my prediction that it becomes
relatively painless to abstract over lists of tokens with the
same tools that work on lists of characters. So, my recent 
exercise shows a tokenizer built using parser combinators
which presents a *token stream* interface which could then
be used by a further layer of parsers.


list token_input( void *s ){
  if(  !s  ) return NULL;
  static parser p;
  if(  !p  ){
    parser space   = many( anyof( " " ) );
    parser digits  = some( is_digit() );
    parser letters = some( is_alpha() );
    p = seq( space, plus( digits, letters ) );
  }
  list r = x( parse( p, s ) );
  return  cons( x( r ), Suspension( xs( r ), token_input ) );
}

void test_tokens(){
  object tokens = token_input( string_input( "123 blah bluh bleh" ) );
  print( take( 2, tokens ) ), puts("");
  print( drop( 2, tokens ) ), puts("");
  print( take( 2, drop( 2, tokens ) ) ), puts("");
}


which has this output:
make -k
cc -std=c99    -c -o pc7.o pc7.c
cc -std=c99  -o pc7_test pc7_test.c pc7.o 
./pc7_test
((() (49 (50 (51 () ) ) ) ) (((32 () ) (98 (108 (97 (104 () ) ) ) ) ) () ) ) 
(((32 () ) (98 (108 (117 (104 () ) ) ) ) ) SUSPENSION ) 
(((32 () ) (98 (108 (117 (104 () ) ) ) ) ) (((32 () ) (98 (108 (101 (104 () ) ) ) ) ) () ) ) 

Which is the desired result. First two tokens extracted on the first
line. Where each token is, so far, just a list of its characters
prepended with any whitespace characters.