Path: csiph.com!1.us.feeder.erje.net!3.us.feeder.erje.net!feeder.erje.net!news.misty.com!news.iecc.com!.POSTED.news.iecc.com!nerds-end
From: Roger L Costello <costello@mitre.org>
Newsgroups: comp.compilers
Subject: An ingenious scanning problem!
Date: Sat, 28 May 2022 19:03:09 +0000
Organization: Compilers Central
Lines: 69
Sender: news@iecc.com
Approved: comp.compilers@iecc.com
Message-ID: <22-05-053@comp.compilers>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 8bit
Injection-Info: gal.iecc.com; posting-host="news.iecc.com:2001:470:1f07:1126:0:676f:7373:6970"; logging-data="42510"; mail-complaints-to="abuse@iecc.com"
Keywords: lex, comment
Posted-Date: 28 May 2022 15:36:40 EDT
X-submission-address: compilers@iecc.com
X-moderator-address: compilers-request@iecc.com
X-FAQ-and-archives: http://compilers.iecc.com
Content-Language: en-US
Xref: csiph.com comp.compilers:3025

Hi Folks,

Page 115-116 of the Flex User's Manual [1] has an ingenious problem posed by
Jan Kort on 05 September 1998.

Here's a description of the problem:

The input contains this: TEST1\n

That is, the input contains the string TEST1 followed by a newline.

The Flex scanner contains a rule that matches on the input:

TEST1\n     { action }

Here's a description of its action:

	Of the text that was matched, I only
	want the first 5 characters (i.e., TEST1).

That action is expressed in Flex this way:

TEST1\n    { yyless(5); }

In the scanner there are also two newline rules.

One newline rule matches on a newline at the beginning of a line (i.e., empty
line):

^\n     { action }

The other newline rule matches on a newline at the end of a line:

\n     { action }

Question: After the action for TEST1\n is executed, which newline rule is
executed?

The answer depends on the meaning of yyless(). Here are two possible meanings
for yyless():

(a) The scanner is backing up in the input stream.
(b) The scanner is pushing new characters onto the input stream.

If the meaning of yyless() is (a), then the second newline rule will be
executed.

If the meaning of yyless() is (b), then the first newline rule will be
executed.

So which does Flex do?

............................

Answer: (b)

Flex treats the newline from TEST1\n as a newline at the beginning of a line
(i.e., empty line). Flex executes this rule:

^\n     { action }

Such an ingenious problem!

Comments?

/Roger

[1] https://epaperpress.com/lexandyacc/download/flex.pdf
[This is a good example of the reasons you don't want your lexer to be too clever. -John]