Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.compilers > #3125

Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple?

Show all headers | View raw

On Thursday, 14 July 2022 at 11:10:56 UTC-4, Roger L Costello wrote:
> [...]
> It seems to me that the lexer should return to the parser the text number
and
> it is the responsibility of the parser to convert the value to an integer
data
> type if it desires.
>
> What do you think?

The division of the job into lexing and parsing is *not* an important
separation of concerns.  Both of these are written at the same time, by the
same person, and the requirements of the parser feed into the lexer in many
detailed ways.  Their specifications are highly coupled and, I expect, almost
always written by the same person pretty much simultaneously.

Instead, this division -- a great big line that divides one part of a
context-free grammar from the other -- is an annoying practical concession
that we make to improve performance in time and size (smaller tables, and
optimization for text), and to get around the limitations in our tools (LR(1)
is a lot less useful when the (1) is a character).

This is specifically why all the answers here are giving you less than
compelling practical justifications for parsing numbers in the lexer and
nobody seems to mind.  There really is no "should" and "should not" w.r.t. the
division between lexing and parsing except what is practical.

Back to comp.compilers | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Roger L Costello <costello@mitre.org> - 2022-07-14 10:25 +0000
  Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? gah4 <gah4@u.washington.edu> - 2022-07-14 10:03 -0700
  Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? George Neuner <gneuner2@comcast.net> - 2022-07-14 16:38 -0400
    Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Spiros Bousbouras <spibou@gmail.com> - 2022-07-15 07:08 +0000
    Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Jan Ziak <0xe2.0x9a.0x9b@gmail.com> - 2022-07-15 03:02 -0700
      Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Jan Ziak <0xe2.0x9a.0x9b@gmail.com> - 2022-07-15 10:50 -0700
      Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? George Neuner <gneuner2@comcast.net> - 2022-07-17 16:52 -0400
        Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Thomas Koenig <tkoenig@netcologne.de> - 2022-07-18 05:44 +0000
      Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? George Neuner <gneuner2@comcast.net> - 2022-07-17 18:01 -0400
  Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Kaz Kylheku <480-992-1380@kylheku.com> - 2022-07-15 14:41 +0000
  Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? "matt.ti...@gmail.com" <matt.timmermans@gmail.com> - 2022-07-16 05:32 -0700
  Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Christopher F Clark <christopher.f.clark@compiler-resources.com> - 2022-07-17 13:10 -0400
    Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? gah4 <gah4@u.washington.edu> - 2022-07-17 20:39 -0700
      Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? gah4 <gah4@u.washington.edu> - 2022-07-19 16:39 -0700
        Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? luser droog <luser.droog@gmail.com> - 2022-07-21 14:16 -0700
      Scannerless parsing was: Why does the lexer convert text integer lexemes ...? "Ev. Drikos" <drikosev@gmail.com> - 2022-07-21 13:41 +0300
        Re: Scannerless parsing was: Why does the lexer convert text integer lexemes ...? "Ev. Drikos" <drikosev@gmail.com> - 2022-07-22 12:29 +0300

csiph-web