Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.compilers > #3126

Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple?

Show key headers only | View raw

You are asking the wrong question.  You are optimizing at the wrong level.
Stop.

Not that long ago, I wrote an article on Quora about this exact phenomenon.

https://www.quora.com/What-do-most-programmers-do-when-optimizing-code-that-is-essentially-wrong/answer/Christopher-F-Clark-1?ch=10&oid=337787257&share=2807f9fb&srid=20qA&target_type=answer

You are focusing on the trivial, the irrelevant.  It is unlikely that
having the lexer convert integers (or floats or quaternions) into a binary
representation is a sufficiently expensive operation to make sense fretting
about it.  And I mean that both in terms of runtime and time spent thinking
about it.  Just do whatever lexer you are following from as an example does
and assume they made the right choice.  The only time you should pay
attention to it, is once you have something working correctly and you have
measured its performance and determined that the code in question is
actually a bottleneck that is significantly impacting performance.  Then,
you have a reason to revisit that choice.

And, when you do, you are as likely to find that the call to atoi is as
much a problem as having the lexer do it.  atoi is going to rescan those
digits causing twice as much work as you would have done by building the
numeric representation while you lexed it.  Of course, that alternative has
its own issues.  It's clearly more complex code in the lexer.

But, again, you shouldn't be worrying about any of that, until you have
isolated it as an actual problem.  Otherwise, figure out something simple
that works.  Having the lexer call atoi seems pretty simple.  It is also
likely to work over a rather large range of inputs.  So, you haven't likely
bought yourself any problems by doing that.

Now, if you are doing SQL (or PL/I) float decimal numbers or handling of
BIGINTs (arbitrarily large integers) or reduced rationals, you might want
something more complex.  But that's isn't the problem in this case.

Don't spend your time worrying about details until you know they are
important details.

Kind regards,
Chris

--
******************************************************************************

Chris Clark                  email:
christopher.f.clark@compiler-resources.com
Compiler Resources, Inc.  Web Site: http://world.std.com/~compres
23 Bailey Rd                 voice: (508) 435-5016
Berlin, MA  01503 USA      twitter: @intel_chris
------------------------------------------------------------------------------

Back to comp.compilers | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Roger L Costello <costello@mitre.org> - 2022-07-14 10:25 +0000
  Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? gah4 <gah4@u.washington.edu> - 2022-07-14 10:03 -0700
  Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? George Neuner <gneuner2@comcast.net> - 2022-07-14 16:38 -0400
    Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Spiros Bousbouras <spibou@gmail.com> - 2022-07-15 07:08 +0000
    Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Jan Ziak <0xe2.0x9a.0x9b@gmail.com> - 2022-07-15 03:02 -0700
      Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Jan Ziak <0xe2.0x9a.0x9b@gmail.com> - 2022-07-15 10:50 -0700
      Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? George Neuner <gneuner2@comcast.net> - 2022-07-17 16:52 -0400
        Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Thomas Koenig <tkoenig@netcologne.de> - 2022-07-18 05:44 +0000
      Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? George Neuner <gneuner2@comcast.net> - 2022-07-17 18:01 -0400
  Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Kaz Kylheku <480-992-1380@kylheku.com> - 2022-07-15 14:41 +0000
  Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? "matt.ti...@gmail.com" <matt.timmermans@gmail.com> - 2022-07-16 05:32 -0700
  Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? Christopher F Clark <christopher.f.clark@compiler-resources.com> - 2022-07-17 13:10 -0400
    Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? gah4 <gah4@u.washington.edu> - 2022-07-17 20:39 -0700
      Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? gah4 <gah4@u.washington.edu> - 2022-07-19 16:39 -0700
        Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple? luser droog <luser.droog@gmail.com> - 2022-07-21 14:16 -0700
      Scannerless parsing was: Why does the lexer convert text integer lexemes ...? "Ev. Drikos" <drikosev@gmail.com> - 2022-07-21 13:41 +0300
        Re: Scannerless parsing was: Why does the lexer convert text integer lexemes ...? "Ev. Drikos" <drikosev@gmail.com> - 2022-07-22 12:29 +0300

csiph-web