Re: Why have a separate lexer instead of parsing 'all the way down'?

Path	csiph.com!aioe.org!.POSTED!not-for-mail
From	"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de>
Newsgroups	comp.programming
Subject	Re: Why have a separate lexer instead of parsing 'all the way down'?
Date	Sun, 29 Jul 2018 10:06:33 +0200
Organization	Aioe.org NNTP Server
Lines	41
Message-ID	<pjjsic$co9$1@gioia.aioe.org> (permalink)
References	<cf70d513-0ca4-4679-9bc9-6fc33a60e557@googlegroups.com>
NNTP-Posting-Host	IzvqdhUtDGKIMCldyDtZ+w.user.gioia.aioe.org
Mime-Version	1.0
Content-Type	text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding	7bit
X-Complaints-To	abuse@aioe.org
User-Agent	Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1
Content-Language	en-US
X-Notice	Filtered by postfilter v. 0.8.3
Xref	csiph.com comp.programming:9065

Show key headers only | View raw

On 2018-07-29 01:11, luser droog wrote:
> I've been doing some scattered readings lately in the realm of parsing
> and the two major examples I've been looking at are Meta II (and TreeMeta)
> which has special handling for terminals like $ID, $ST (string), and $NUM,
> ie. isolated lexical analysis functions; and parser combinators which
> (in the examples I've seen) just use parsers all the way down, ie. the
> lexical analysis of building tokens uses the same mechanisms as the
> syntactic analysis of the token arrangement uses.
> 
> What are the considerations for preferring to have a separate lexer or not?
> It seems /parsimonious/ (!) to use parsers all the way down, but lexing is
> "a simpler job" so it also makes sense to wish for a more specialized tool.
> Does the difference depend upon the details of the language you wish to
> parse? Is the decision different if there are 2 or 3 types of tokens vs.
> 30 to 40?
> 
> I'd be interested in hearing any thoughts one way or the other, hoping that
> the topic is sufficiently esoteric that flame wars are unlikely and any
> strongly held opinions have at least some entertaining anecdote behind them.

Of course there will be flame wars. There are people who like formal 
grammars who consequently like lexers. And there are people like me who 
consider grammars useless and see no technical reason to have lexers, 
tokenizers etc, while see many good reasons not to have this mess from 
the software architecture POV.

Software techniques hugely evolved since 70's. We know how to go to the 
semantic analysis in just one hop. It is not magic not even rocket 
science anymore.

P.S. The only valid argument for formal grammars may be the issue of 
formal verification and proof. The problem with that is that the formal 
grammar language is far more complex for a human reader than any decent 
general purpose programming language. So the chance to get errors in the 
formal specification (grammar) is higher than for a hand-written 
implementation of a parser.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Thread

Why have a separate lexer instead of parsing 'all the way down'? luser droog <mijoryx@yahoo.com> - 2018-07-28 16:11 -0700
  Re: Why have a separate lexer instead of parsing 'all the way down'? "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> - 2018-07-29 10:06 +0200
  Re: Why have a separate lexer instead of parsing 'all the way down'? Richard Heathfield <rjh@cpax.org.uk> - 2018-07-29 12:19 +0100
    Re: Why have a separate lexer instead of parsing 'all the way down'? Chad <cdalten@gmail.com> - 2018-07-30 09:02 -0700

csiph-web