Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #143041 > unrolled thread

[PC] semi-lazy tokenizer

Back to article view | Back to comp.lang.c

Contents

  [PC] semi-lazy tokenizer M Joshua Ryan <luser.droog@gmail.com> - 2019-04-11 16:56 -0500
    Re: [PC] semi-lazy tokenizer Chad <cdalten@gmail.com> - 2019-04-11 15:18 -0700
    Re: [PC] semi-lazy tokenizer M Joshua Ryan <luser.droog@gmail.com> - 2019-04-13 00:07 -0500
      Re: [PC] semi-lazy tokenizer fir <profesor.fir@gmail.com> - 2019-04-14 12:21 -0700
        Re: [PC] semi-lazy tokenizer fir <profesor.fir@gmail.com> - 2019-04-14 12:47 -0700
        Re: [PC] semi-lazy tokenizer luser droog <luser.droog@gmail.com> - 2019-04-14 23:05 -0700
          Re: [PC] semi-lazy tokenizer fir <profesor.fir@gmail.com> - 2019-04-15 00:26 -0700
            Re: [PC] semi-lazy tokenizer fir <profesor.fir@gmail.com> - 2019-04-15 00:51 -0700
              Re: [PC] semi-lazy tokenizer fir <profesor.fir@gmail.com> - 2019-04-15 07:06 -0700
                Re: [PC] semi-lazy tokenizer fir <profesor.fir@gmail.com> - 2019-04-15 07:13 -0700

#143041 — [PC] semi-lazy tokenizer

As discovered by Ben in a previous thread, I'm not as lazy as
I could be (but expending great effort to become so). But even
with half-lazy the code has met my prediction that it becomes
relatively painless to abstract over lists of tokens with the
same tools that work on lists of characters. So, my recent 
exercise shows a tokenizer built using parser combinators
which presents a *token stream* interface which could then
be used by a further layer of parsers.


list token_input( void *s ){
  if(  !s  ) return NULL;
  static parser p;
  if(  !p  ){
    parser space   = many( anyof( " " ) );
    parser digits  = some( is_digit() );
    parser letters = some( is_alpha() );
    p = seq( space, plus( digits, letters ) );
  }
  list r = x( parse( p, s ) );
  return  cons( x( r ), Suspension( xs( r ), token_input ) );
}

void test_tokens(){
  object tokens = token_input( string_input( "123 blah bluh bleh" ) );
  print( take( 2, tokens ) ), puts("");
  print( drop( 2, tokens ) ), puts("");
  print( take( 2, drop( 2, tokens ) ) ), puts("");
}


which has this output:
make -k
cc -std=c99    -c -o pc7.o pc7.c
cc -std=c99  -o pc7_test pc7_test.c pc7.o 
./pc7_test
((() (49 (50 (51 () ) ) ) ) (((32 () ) (98 (108 (97 (104 () ) ) ) ) ) () ) ) 
(((32 () ) (98 (108 (117 (104 () ) ) ) ) ) SUSPENSION ) 
(((32 () ) (98 (108 (117 (104 () ) ) ) ) ) (((32 () ) (98 (108 (101 (104 () ) ) ) ) ) () ) ) 

Which is the desired result. First two tokens extracted on the first
line. Where each token is, so far, just a list of its characters
prepended with any whitespace characters.

[toc] | [next] | [standalone]

#143043

On Thursday, April 11, 2019 at 2:56:52 PM UTC-7, luser droog wrote:
> As discovered by Ben in a previous thread, I'm not as lazy as
> I could be (but expending great effort to become so). But even
> with half-lazy the code has met my prediction that it becomes
> relatively painless to abstract over lists of tokens with the
> same tools that work on lists of characters. So, my recent 
> exercise shows a tokenizer built using parser combinators
> which presents a *token stream* interface which could then
> be used by a further layer of parsers.
> 
> 
> list token_input( void *s ){
>   if(  !s  ) return NULL;
>   static parser p;
>   if(  !p  ){
>     parser space   = many( anyof( " " ) );
>     parser digits  = some( is_digit() );
>     parser letters = some( is_alpha() );
>     p = seq( space, plus( digits, letters ) );
>   }
>   list r = x( parse( p, s ) );
>   return  cons( x( r ), Suspension( xs( r ), token_input ) );
> }
> 
> void test_tokens(){
>   object tokens = token_input( string_input( "123 blah bluh bleh" ) );
>   print( take( 2, tokens ) ), puts("");
>   print( drop( 2, tokens ) ), puts("");
>   print( take( 2, drop( 2, tokens ) ) ), puts("");
> }
> 
> 
> which has this output:
> make -k
> cc -std=c99    -c -o pc7.o pc7.c
> cc -std=c99  -o pc7_test pc7_test.c pc7.o 
> ./pc7_test
> ((() (49 (50 (51 () ) ) ) ) (((32 () ) (98 (108 (97 (104 () ) ) ) ) ) () ) ) 
> (((32 () ) (98 (108 (117 (104 () ) ) ) ) ) SUSPENSION ) 
> (((32 () ) (98 (108 (117 (104 () ) ) ) ) ) (((32 () ) (98 (108 (101 (104 () ) ) ) ) ) () ) ) 
> 
> Which is the desired result. First two tokens extracted on the first
> line. Where each token is, so far, just a list of its characters
> prepended with any whitespace characters.

Well, I know a Ph.D who is an expert in this kind of stuff. I should refer you to this guy because he is totally awesome. He's proud of the fact that approximately 40% of the undergraduate students in his one class are taking it for the second time because he failed them the first time around.

And his one and only superstar student got rejected by Stanford and now works on  a farm south of Dallas, Texas.

[toc] | [prev] | [next] | [standalone]

#143066

M Joshua Ryan <luser.droog@gmail.com> writes:

> As discovered by Ben in a previous thread, I'm not as lazy as
> I could be (but expending great effort to become so). But even
> with half-lazy the code has met my prediction that it becomes
> relatively painless to abstract over lists of tokens with the
> same tools that work on lists of characters. So, my recent 
> exercise shows a tokenizer built using parser combinators
> which presents a *token stream* interface which could then
> be used by a further layer of parsers.
>

Some further enhancements make the output nicer and the results
more useful. I've added functions to convert a list of characters
into a string or and integer. And a constructor for "symbols" 
which map to enum values (but also have a printable name string).

Hooking these into the tokenizer looks like this:


enum extended_symbols { t_int = USER_SYMBOL, t_num };

object on_digits( void *v, list o ){ return  cons( Symbol(t_int), to_integer( o ) ); }
object on_letters( void *v, list o ){ return  cons( Symbol(t_num), to_string( o ) ); }
list token_input( void *s ){
  if(  !s  ) return NULL;
  static parser p;
  if(  !p  ){
    parser space   = many( anyof( " " ) );
    parser digits  = using( some( is_digit() ), on_digits  );
    parser letters = using( some( is_alpha() ), on_letters );
    p = seq( space, plus( digits, letters ) );
  }
  list r = x( parse( p, s ) );
  return  cons( x( r ), Suspension( xs( r ), token_input ) );
}


And produces this (nicer) output:

make -k
cc -std=c99    -c -o pc7.o pc7.c
cc -std=c99  -o pc7_test pc7_test.c pc7.o 
./pc7_test
((() (t_int 123 ) ) (((32 () ) (t_num "blah" ) ) () ) ) 
(((32 () ) (t_num "bluh" ) ) SUSPENSION ) 
(((32 () ) (t_num "bluh" ) ) (((32 () ) (t_num "bleh" ) ) () ) ) 


The next level of parsers can operate on these "token objects" 
by defining predicates and using the 'parser sat( predicate pred )' 
parser constructor which is type agnostic as far as the elements
of the input list. 

I suppose the next step is to try a real grammar, maybe an early
version of C...

[toc] | [prev] | [next] | [standalone]

#143077

W dniu sobota, 13 kwietnia 2019 07:07:24 UTC+2 użytkownik luser droog napisał:
> 
> I suppose the next step is to try a real grammar, maybe an early
> version of C...

when i was starting to write my proto c (as for now) compiler (wrote some now i get a break) my ovn discovery was c dont needs
any kind of elaborate 'tokenizer' - anough is to use smething that will just index even as primitive atoms like 1) alphanumeric labels 2) pack of numeric digits 3) operator-like characters (and just 4) filter out spaces) then use just a bunch of customized switches, and thats all


if someone want to use more 'eleborate' tokenizer i wouldnt stop him but imo its rather not optimal 


btw whan i wrote it (dont know 2 months before) i was thinking that this "atomizer" shouldnt at least extract a bit more elaborat things i mean extract '-2' as a sperate atom not two atoms '-' and '2' as this more primitive one, or extract "asdkh as;dhja dslkjh" also as one instead of 7 as in this primitive 
case, but now im like i dont even care
(now call that a 'lazy' though to ne honest its even maybe more 'untidy'/'careless' than lazy)

[toc] | [prev] | [next] | [standalone]

#143078

W dniu niedziela, 14 kwietnia 2019 21:21:19 UTC+2 użytkownik fir napisał:
> W dniu sobota, 13 kwietnia 2019 07:07:24 UTC+2 użytkownik luser droog napisał:
> > 
> > I suppose the next step is to try a real grammar, maybe an early
> > version of C...
> 
> when i was starting to write my proto c (as for now) compiler (wrote some now i get a break) my ovn discovery was c dont needs
> any kind of elaborate 'tokenizer' - anough is to use smething that will just index even as primitive atoms like 1) alphanumeric labels 2) pack of numeric digits 3) operator-like characters (and just 4) filter out spaces) then use just a bunch of customized switches, and thats all
> 
> 
> if someone want to use more 'eleborate' tokenizer i wouldnt stop him but imo its rather not optimal 
> 
> 
> btw whan i wrote it (dont know 2 months before) i was thinking that this "atomizer" shouldnt at least extract a bit more elaborat things i mean extract '-2' as a sperate atom not two atoms '-' and '2' as this more primitive one, or extract "asdkh as;dhja dslkjh" also as one instead of 7 as in this primitive 
> case, but now im like i dont even care
> (now call that a 'lazy' though to ne honest its even maybe more 'untidy'/'careless' than lazy)

obviously carelessnes ofers a big energy saving 
carelessnes posibbly even more than laziness

and im not sure but maybe i could say with time i get more and more careless

(and i dont find it bad its maybe more liek im not sure what to think on it 

i could say carelessnes is good (in coding) 

but unfortunatelly there is paralel thing of losing motivation for goals too - (though im not strictly sure if they are related, they may be somewhat related though maybe not quite strictly) so im overally not sure

[toc] | [prev] | [next] | [standalone]

#143080

On Sunday, April 14, 2019 at 2:21:19 PM UTC-5, fir wrote:
> W dniu sobota, 13 kwietnia 2019 07:07:24 UTC+2 użytkownik luser droog napisał:
> > 
> > I suppose the next step is to try a real grammar, maybe an early
> > version of C...
> 
> when i was starting to write my proto c (as for now) compiler (wrote some now i get a break) my ovn discovery was c dont needs
> any kind of elaborate 'tokenizer' - anough is to use smething that will just index even as primitive atoms like 1) alphanumeric labels 2) pack of numeric digits 3) operator-like characters (and just 4) filter out spaces) then use just a bunch of customized switches, and thats all
> 
> 
> if someone want to use more 'eleborate' tokenizer i wouldnt stop him but imo its rather not optimal 
> 
> 
> btw whan i wrote it (dont know 2 months before) i was thinking that this "atomizer" shouldnt at least extract a bit more elaborat things i mean extract '-2' as a sperate atom not two atoms '-' and '2' as this more primitive one, or extract "asdkh as;dhja dslkjh" also as one instead of 7 as in this primitive 
> case, but now im like i dont even care
> (now call that a 'lazy' though to ne honest its even maybe more 'untidy'/'careless' than lazy)

[answering in middle-endian order]
No idea what you're talking about. Laziness means not executing a function
right now, but saving it and calling it later if needed.

We already covered the minus sign. It's an operator in C, all integers are
positive. finito. 

The parser combinator code could be considered *elaborate*, but each piece
is *really really small* and should be relatively easy to understand. But
there is also a /sense/ behind it, or Gestalt which if not grokked might
make the whole thing impenetrable. I keep meaning to write everything up
with a breakdown in a sort of literate programming style to get everyone
up to speed. As it is, I think Ben and Tim have some idea of what I'm doing
but possibly few others. That first code dump probably could have used a
comment or two. :)

[toc] | [prev] | [next] | [standalone]

#143081

W dniu poniedziałek, 15 kwietnia 2019 08:05:46 UTC+2 użytkownik luser droog napisał:
> On Sunday, April 14, 2019 at 2:21:19 PM UTC-5, fir wrote:
> > W dniu sobota, 13 kwietnia 2019 07:07:24 UTC+2 użytkownik luser droog napisał:
> > > 
> > > I suppose the next step is to try a real grammar, maybe an early
> > > version of C...
> > 
> > when i was starting to write my proto c (as for now) compiler (wrote some now i get a break) my ovn discovery was c dont needs
> > any kind of elaborate 'tokenizer' - anough is to use smething that will just index even as primitive atoms like 1) alphanumeric labels 2) pack of numeric digits 3) operator-like characters (and just 4) filter out spaces) then use just a bunch of customized switches, and thats all
> > 
> > 
> > if someone want to use more 'eleborate' tokenizer i wouldnt stop him but imo its rather not optimal 
> > 
> > 
> > btw whan i wrote it (dont know 2 months before) i was thinking that this "atomizer" shouldnt at least extract a bit more elaborat things i mean extract '-2' as a sperate atom not two atoms '-' and '2' as this more primitive one, or extract "asdkh as;dhja dslkjh" also as one instead of 7 as in this primitive 
> > case, but now im like i dont even care
> > (now call that a 'lazy' though to ne honest its even maybe more 'untidy'/'careless' than lazy)
> 
> [answering in middle-endian order]
> No idea what you're talking about. Laziness means not executing a function
> right now, but saving it and calling it later if needed.
> 
> We already covered the minus sign. It's an operator in C, all integers are
> positive. finito. 
> 
> The parser combinator code could be considered *elaborate*, but each piece
> is *really really small* and should be relatively easy to understand. But
> there is also a /sense/ behind it, or Gestalt which if not grokked might
> make the whole thing impenetrable. I keep meaning to write everything up
> with a breakdown in a sort of literate programming style to get everyone
> up to speed. As it is, I think Ben and Tim have some idea of what I'm doing
> but possibly few others. That first code dump probably could have used a
> comment or two. :)

lol, (and yawn)

[toc] | [prev] | [next] | [standalone]

#143082

W dniu poniedziałek, 15 kwietnia 2019 09:26:44 UTC+2 użytkownik fir napisał:
> W dniu poniedziałek, 15 kwietnia 2019 08:05:46 UTC+2 użytkownik luser droog napisał:
> > On Sunday, April 14, 2019 at 2:21:19 PM UTC-5, fir wrote:
> > > W dniu sobota, 13 kwietnia 2019 07:07:24 UTC+2 użytkownik luser droog napisał:
> > > > 
> > > > I suppose the next step is to try a real grammar, maybe an early
> > > > version of C...
> > > 
> > > when i was starting to write my proto c (as for now) compiler (wrote some now i get a break) my ovn discovery was c dont needs
> > > any kind of elaborate 'tokenizer' - anough is to use smething that will just index even as primitive atoms like 1) alphanumeric labels 2) pack of numeric digits 3) operator-like characters (and just 4) filter out spaces) then use just a bunch of customized switches, and thats all
> > > 
> > > 
> > > if someone want to use more 'eleborate' tokenizer i wouldnt stop him but imo its rather not optimal 
> > > 
> > > 
> > > btw whan i wrote it (dont know 2 months before) i was thinking that this "atomizer" shouldnt at least extract a bit more elaborat things i mean extract '-2' as a sperate atom not two atoms '-' and '2' as this more primitive one, or extract "asdkh as;dhja dslkjh" also as one instead of 7 as in this primitive 
> > > case, but now im like i dont even care
> > > (now call that a 'lazy' though to ne honest its even maybe more 'untidy'/'careless' than lazy)
> > 
> > [answering in middle-endian order]
> > No idea what you're talking about. Laziness means not executing a function
> > right now, but saving it and calling it later if needed.
> > 
> > We already covered the minus sign. It's an operator in C, all integers are
> > positive. finito. 
> > 
> > The parser combinator code could be considered *elaborate*, but each piece
> > is *really really small* and should be relatively easy to understand. But
> > there is also a /sense/ behind it, or Gestalt which if not grokked might
> > make the whole thing impenetrable. I keep meaning to write everything up
> > with a breakdown in a sort of literate programming style to get everyone
> > up to speed. As it is, I think Ben and Tim have some idea of what I'm doing
> > but possibly few others. That first code dump probably could have used a
> > comment or two. :)
> 
> lol, (and yawn)

yawning becouse that answer was not interesting - and sorta stupid to be honest
(also yawning becouse i dint sleep long again, sadly)

what i could add? when i wrote proto compiler the atomizer (name it tokenizer if you want/must) was not really part of any more significant work (it is indeed easy small and 'automatic') the more problem is what i call 'switches' (those parts that 
reckognize different types of expressions 
in c syntax and generate code for that)

its also not hard but its physically much more (relatively easy) matter to think up

i could say (maybe a bit unprecisely) that when you want to code that first pass - i mean splitter+atomiser and setup all project to check how it began to work and code some basics it will take 9for me) 3 days of work.. then t code those switch (or switchers) would be more like next 27 days making it all a month of coding  maybe

i mean switchers are more real matter... if i rest maybe i will write somethiing on 'em

[toc] | [prev] | [next] | [standalone]

#143084

W dniu poniedziałek, 15 kwietnia 2019 09:51:54 UTC+2 użytkownik fir napisał:
> W dniu poniedziałek, 15 kwietnia 2019 09:26:44 UTC+2 użytkownik fir napisał:
> > W dniu poniedziałek, 15 kwietnia 2019 08:05:46 UTC+2 użytkownik luser droog napisał:
> > > On Sunday, April 14, 2019 at 2:21:19 PM UTC-5, fir wrote:
> > > > W dniu sobota, 13 kwietnia 2019 07:07:24 UTC+2 użytkownik luser droog napisał:
> > > > > 
> > > > > I suppose the next step is to try a real grammar, maybe an early
> > > > > version of C...
> > > > 
> > > > when i was starting to write my proto c (as for now) compiler (wrote some now i get a break) my ovn discovery was c dont needs
> > > > any kind of elaborate 'tokenizer' - anough is to use smething that will just index even as primitive atoms like 1) alphanumeric labels 2) pack of numeric digits 3) operator-like characters (and just 4) filter out spaces) then use just a bunch of customized switches, and thats all
> > > > 
> > > > 
> > > > if someone want to use more 'eleborate' tokenizer i wouldnt stop him but imo its rather not optimal 
> > > > 
> > > > 
> > > > btw whan i wrote it (dont know 2 months before) i was thinking that this "atomizer" shouldnt at least extract a bit more elaborat things i mean extract '-2' as a sperate atom not two atoms '-' and '2' as this more primitive one, or extract "asdkh as;dhja dslkjh" also as one instead of 7 as in this primitive 
> > > > case, but now im like i dont even care
> > > > (now call that a 'lazy' though to ne honest its even maybe more 'untidy'/'careless' than lazy)
> > > 
> > > [answering in middle-endian order]
> > > No idea what you're talking about. Laziness means not executing a function
> > > right now, but saving it and calling it later if needed.
> > > 
> > > We already covered the minus sign. It's an operator in C, all integers are
> > > positive. finito. 
> > > 
> > > The parser combinator code could be considered *elaborate*, but each piece
> > > is *really really small* and should be relatively easy to understand. But
> > > there is also a /sense/ behind it, or Gestalt which if not grokked might
> > > make the whole thing impenetrable. I keep meaning to write everything up
> > > with a breakdown in a sort of literate programming style to get everyone
> > > up to speed. As it is, I think Ben and Tim have some idea of what I'm doing
> > > but possibly few others. That first code dump probably could have used a
> > > comment or two. :)
> > 
> > lol, (and yawn)
> 
> yawning becouse that answer was not interesting - and sorta stupid to be honest
> (also yawning becouse i dint sleep long again, sadly)
> 
> what i could add? when i wrote proto compiler the atomizer (name it tokenizer if you want/must) was not really part of any more significant work (it is indeed easy small and 'automatic') the more problem is what i call 'switches' (those parts that 
> reckognize different types of expressions 
> in c syntax and generate code for that)
> 
> its also not hard but its physically much more (relatively easy) matter to think up
> 
> i could say (maybe a bit unprecisely) that when you want to code that first pass - i mean splitter+atomiser and setup all project to check how it began to work and code some basics it will take 9for me) 3 days of work.. then t code those switch (or switchers) would be more like next 27 days making it all a month of coding  maybe
> 
> i mean switchers are more real matter... if i rest maybe i will write somethiing on 'em

i may add tat talking on this 'tokenizer' to much is like botah bloat and bad distraction IMO and that is (partially on what i said)

its a bit like making a whole screen of code 
to set a pixel on screen instead of calling SetPixel(100,100, 0x008000)  (todays opengl is much worse than one screen) and talkin on 
this 'tokenizer' is like that though its more 'theoretical' bloat imo (when when i write it normal way its like two functions 
split() and atomize() and none of those 2 is complex

ant there come those 'switchers' as i called em - and that is more substantiall part

(hovever to write more clear on them i should wrote them all, and i yet not wrote them all, only some)

but besides wrote as you like, i only just say imo its rather kinda bloat - and especially i cant agree if someone calls this bloat right way especially 'only right way' (besides i dont bant to engage in any vulgar and nonsense (bartc-style) fights on it, only share some views on it, thats all)

[toc] | [prev] | [next] | [standalone]

#143085

W dniu poniedziałek, 15 kwietnia 2019 16:07:02 UTC+2 użytkownik fir napisał:
> W dniu poniedziałek, 15 kwietnia 2019 09:51:54 UTC+2 użytkownik fir napisał:
> > W dniu poniedziałek, 15 kwietnia 2019 09:26:44 UTC+2 użytkownik fir napisał:
> > > W dniu poniedziałek, 15 kwietnia 2019 08:05:46 UTC+2 użytkownik luser droog napisał:
> > > > On Sunday, April 14, 2019 at 2:21:19 PM UTC-5, fir wrote:
> > > > > W dniu sobota, 13 kwietnia 2019 07:07:24 UTC+2 użytkownik luser droog napisał:
> > > > > > 
> > > > > > I suppose the next step is to try a real grammar, maybe an early
> > > > > > version of C...
> > > > > 
> > > > > when i was starting to write my proto c (as for now) compiler (wrote some now i get a break) my ovn discovery was c dont needs
> > > > > any kind of elaborate 'tokenizer' - anough is to use smething that will just index even as primitive atoms like 1) alphanumeric labels 2) pack of numeric digits 3) operator-like characters (and just 4) filter out spaces) then use just a bunch of customized switches, and thats all
> > > > > 
> > > > > 
> > > > > if someone want to use more 'eleborate' tokenizer i wouldnt stop him but imo its rather not optimal 
> > > > > 
> > > > > 
> > > > > btw whan i wrote it (dont know 2 months before) i was thinking that this "atomizer" shouldnt at least extract a bit more elaborat things i mean extract '-2' as a sperate atom not two atoms '-' and '2' as this more primitive one, or extract "asdkh as;dhja dslkjh" also as one instead of 7 as in this primitive 
> > > > > case, but now im like i dont even care
> > > > > (now call that a 'lazy' though to ne honest its even maybe more 'untidy'/'careless' than lazy)
> > > > 
> > > > [answering in middle-endian order]
> > > > No idea what you're talking about. Laziness means not executing a function
> > > > right now, but saving it and calling it later if needed.
> > > > 
> > > > We already covered the minus sign. It's an operator in C, all integers are
> > > > positive. finito. 
> > > > 
> > > > The parser combinator code could be considered *elaborate*, but each piece
> > > > is *really really small* and should be relatively easy to understand. But
> > > > there is also a /sense/ behind it, or Gestalt which if not grokked might
> > > > make the whole thing impenetrable. I keep meaning to write everything up
> > > > with a breakdown in a sort of literate programming style to get everyone
> > > > up to speed. As it is, I think Ben and Tim have some idea of what I'm doing
> > > > but possibly few others. That first code dump probably could have used a
> > > > comment or two. :)
> > > 
> > > lol, (and yawn)
> > 
> > yawning becouse that answer was not interesting - and sorta stupid to be honest
> > (also yawning becouse i dint sleep long again, sadly)
> > 
> > what i could add? when i wrote proto compiler the atomizer (name it tokenizer if you want/must) was not really part of any more significant work (it is indeed easy small and 'automatic') the more problem is what i call 'switches' (those parts that 
> > reckognize different types of expressions 
> > in c syntax and generate code for that)
> > 
> > its also not hard but its physically much more (relatively easy) matter to think up
> > 
> > i could say (maybe a bit unprecisely) that when you want to code that first pass - i mean splitter+atomiser and setup all project to check how it began to work and code some basics it will take 9for me) 3 days of work.. then t code those switch (or switchers) would be more like next 27 days making it all a month of coding  maybe
> > 
> > i mean switchers are more real matter... if i rest maybe i will write somethiing on 'em
> 
> i may add tat talking on this 'tokenizer' to much is like botah bloat and bad distraction IMO and that is (partially on what i said)
> 
> its a bit like making a whole screen of code 
> to set a pixel on screen instead of calling SetPixel(100,100, 0x008000)  (todays opengl is much worse than one screen) and talkin on 
> this 'tokenizer' is like that though its more 'theoretical' bloat imo (when when i write it normal way its like two functions 
> split() and atomize() and none of those 2 is complex
> 
> ant there come those 'switchers' as i called em - and that is more substantiall part
> 
> (hovever to write more clear on them i should wrote them all, and i yet not wrote them all, only some)
> 
> but besides wrote as you like, i only just say imo its rather kinda bloat - and especially i cant agree if someone calls this bloat right way especially 'only right way' (besides i dont bant to engage in any vulgar and nonsense (bartc-style) fights on it, only share some views on it, thats all)

besides i find your post quite ok and on topic i only think

1) if you do specific things (i mean specific things in specific way) you possibly shuld add some easy introduction
to it - just to make reader understending better on what you wrote

in other case the reader would need to heavuily guess what you really do and most people wouldnt want to engage in that guessing and then it has no answers

2) dont post to much threads in short period of time (i think some amount like 5 (or some more but not much more, line not more then 10) new therads in month should be reasoneble

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.c

csiph-web