Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!usenet.blueworldhosting.com!feeder01.blueworldhosting.com!border2.nntp.dca1.giganews.com!nntp.giganews.com!news.iecc.com!.POSTED!nerds-end From: Evangelos Drikos Newsgroups: comp.compilers Subject: Re: split tokens; was LR(1) Parsing : Error Handling & Recovery Date: Wed, 03 Sep 2014 16:57:45 +0300 Organization: An OTEnet S.A. customer Lines: 35 Sender: news@iecc.com Approved: comp.compilers@iecc.com Message-ID: <14-09-001@comp.compilers> References: <14-07-023@comp.compilers> <14-07-024@comp.compilers> <14-07-030@comp.compilers> <14-07-031@comp.compilers> <14-07-038@comp.compilers> <14-07-039@comp.compilers> <14-07-040@comp.compilers> <14-07-049@comp.compilers> <14-07-051@comp.compilers> <14-08-009@comp.compilers> <14-08-010@comp.compilers> NNTP-Posting-Host: news.iecc.com Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Trace: miucha.iecc.com 1409752968 45082 2001:470:1f07:1126:0:676f:7373:6970 (3 Sep 2014 14:02:48 GMT) X-Complaints-To: abuse@iecc.com NNTP-Posting-Date: Wed, 3 Sep 2014 14:02:48 +0000 (UTC) Keywords: parse, bison Posted-Date: 03 Sep 2014 10:02:48 EDT X-submission-address: compilers@iecc.com X-moderator-address: compilers-request@iecc.com X-FAQ-and-archives: http://compilers.iecc.com Xref: csiph.com comp.compilers:1367 On 8/29/14 12:13 PM, Hans Aberg wrote: > On 2014/08/26 12:44, Evangelos Drikos wrote: >> ... What the Bison team said about your idea? >>... > ... we discussed split tokens, which it cannot currently handle > ... The main focus is on computer languages, though it > would be nice to be able to experiment with natural languages. Split tokens are also useful for computer languages like SQL which have unreserved keywords; yet, an efficient solution requires enough code. Bison & Flex are fast and mature tools that can be combined efficiently in a simple and straightforward manner for some computer languages. But, Bison's GLR probably cannot parse some context free grammars. One such epsilon grammar given by Nozohoor-Farshi[NF91] has broken Tomita's original algorithm; a slightly modified grammar for Bison is given here: G1: S ; S: A S 'b' | 'x' ; A: {printf("Delayed forever!\n");} ; Given the string "xbbb", a Bison (2.3) GLR parser created by the above grammar goes into an infinitive loop. Perhaps, this well known issue is either solved with a hack or not perceived as a bug exactly because the main focus is on computer languages. Probably, one could find a plethora of other open source tools for NLP. Regards, Ev. Drikos