Path: csiph.com!xmission!news.snarked.org!news.linkpendium.com!news.linkpendium.com!news.iecc.com!.POSTED.news.iecc.com!nerds-end From: Maury Markowitz Newsgroups: comp.compilers Subject: Re: Languages with optional spaces Date: Tue, 25 Feb 2020 06:11:18 -0800 (PST) Organization: Compilers Central Lines: 10 Sender: news@iecc.com Approved: comp.compilers@iecc.com Message-ID: <20-02-018@comp.compilers> References: <20-02-015@comp.compilers> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Injection-Info: gal.iecc.com; posting-host="news.iecc.com:2001:470:1f07:1126:0:676f:7373:6970"; logging-data="43974"; mail-complaints-to="abuse@iecc.com" Keywords: Basic, lex, comment Posted-Date: 25 Feb 2020 10:47:06 EST X-submission-address: compilers@iecc.com X-moderator-address: compilers-request@iecc.com X-FAQ-and-archives: http://compilers.iecc.com In-Reply-To: <20-02-015@comp.compilers> Xref: csiph.com comp.compilers:2456 So it seems the most basic answer is "no, it won't do that". Perhaps this should be a feature? "Tokens are immediate terminals" or somesuch. I originally chose flex/bison because it is widely available on almost any platform (although on Windows it is sub-optimal) and I found a reasonable BASIC starting point online. In any event, it does seem the practical solution is to make my own lexer. Luckily there are any number that exist already in cross-platform form. Perhaps you all have some starting points I should look at? I'm most interested in plain C although I am definitely going to look at Jerry's code. [Writing a lexer for Basic shouldn't be very hard since the token structure is pretty straightforward. The hardest part is being sure you detect and deal with invalid input in all the places it can occur. C lexers tend to be loops with big switch statements on the input characters, and you need to remember to have a default in all of them. -John]