Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.compilers > #3129

Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple?

Show all headers | View raw

On Sunday, July 17, 2022 at 10:29:53 AM UTC-7, christoph...@compiler-resources.com wrote:
> You are asking the wrong question. You are optimizing at the wrong level.
> Stop.

(snip)

> You are focusing on the trivial, the irrelevant. It is unlikely that
> having the lexer convert integers (or floats or quaternions) into a binary
> representation is a sufficiently expensive operation to make sense fretting
> about it.

Not so long ago, I wrote about the whole idea of separating lexing and parsing
was overoptimizing.  It was needed many years ago, when computer memories
were in kilobytes, not yet megabytes or gigabytes.  But got almost no comment.

Converting to integer has lots of problems, especially in the case of overflow.

In some cases, and even for some people, separating lexing and parsing might
make it easier to write and/or understand. Those are important.

But maybe not.

The STEP processor, which I have written about before, mostly doesn't
separate them.  There are some built-in rules related to where macro
matches can start and end, and those are similar to the results of
some lexical analysis.  Some of those rules speed up processing, which
was more important 45 years ago than now.

I suspect that I believe now that they could be separate, but implemented
by the same program.  That would allow them to be separated where
convenient for the user, and also not separated where it is easier.
[In my experience separating the lexer from the parser makes it a lot easier
to deal with common lexical situations like skipping white space and comments.
You could certainly do that in a combined scheme but I'm not sure it would end
up any simpler. -John]

Back to comp.compilers | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Roger L Costello <costello@mitre.org> - 2022-07-14 10:25 +0000
  Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? gah4 <gah4@u.washington.edu> - 2022-07-14 10:03 -0700
  Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? George Neuner <gneuner2@comcast.net> - 2022-07-14 16:38 -0400
    Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Spiros Bousbouras <spibou@gmail.com> - 2022-07-15 07:08 +0000
    Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Jan Ziak <0xe2.0x9a.0x9b@gmail.com> - 2022-07-15 03:02 -0700
      Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Jan Ziak <0xe2.0x9a.0x9b@gmail.com> - 2022-07-15 10:50 -0700
      Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? George Neuner <gneuner2@comcast.net> - 2022-07-17 16:52 -0400
        Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Thomas Koenig <tkoenig@netcologne.de> - 2022-07-18 05:44 +0000
      Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? George Neuner <gneuner2@comcast.net> - 2022-07-17 18:01 -0400
  Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Kaz Kylheku <480-992-1380@kylheku.com> - 2022-07-15 14:41 +0000
  Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? "matt.ti...@gmail.com" <matt.timmermans@gmail.com> - 2022-07-16 05:32 -0700
  Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Christopher F Clark <christopher.f.clark@compiler-resources.com> - 2022-07-17 13:10 -0400
    Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? gah4 <gah4@u.washington.edu> - 2022-07-17 20:39 -0700
      Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? gah4 <gah4@u.washington.edu> - 2022-07-19 16:39 -0700
        Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? luser droog <luser.droog@gmail.com> - 2022-07-21 14:16 -0700
      Scannerless parsing was: Why does the lexer convert text integer lexemes ...? "Ev. Drikos" <drikosev@gmail.com> - 2022-07-21 13:41 +0300
        Re: Scannerless parsing was: Why does the lexer convert text integer lexemes ...? "Ev. Drikos" <drikosev@gmail.com> - 2022-07-22 12:29 +0300

csiph-web