Path: csiph.com!x330-a1.tempe.blueboxinc.net!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!npeer02.iad.highwinds-media.com!news.highwinds-media.com!feed-me.highwinds-media.com!nx01.iad01.newshosting.com!newshosting.com!216.196.98.146.MISMATCH!border3.nntp.dca.giganews.com!border1.nntp.dca.giganews.com!nntp.giganews.com!news.iecc.com!nerds-end From: Andrew Tomazos Newsgroups: comp.compilers Subject: Re: Expected Token Density in Random Stream Date: Tue, 13 Dec 2011 06:00:40 -0800 (PST) Organization: Compilers Central Lines: 17 Sender: news@iecc.com Approved: comp.compilers@iecc.com Message-ID: <11-12-019@comp.compilers> References: <11-12-015@comp.compilers> <11-12-016@comp.compilers> NNTP-Posting-Host: news.iecc.com X-Trace: leila.iecc.com 1323887306 33017 64.57.183.58 (14 Dec 2011 18:28:26 GMT) X-Complaints-To: abuse@iecc.com NNTP-Posting-Date: Wed, 14 Dec 2011 18:28:26 +0000 (UTC) Keywords: parse, theory Posted-Date: 14 Dec 2011 13:28:26 EST X-submission-address: compilers@iecc.com X-moderator-address: compilers-request@iecc.com X-FAQ-and-archives: http://compilers.iecc.com Xref: x330-a1.tempe.blueboxinc.net comp.compilers:391 On Dec 11, 6:56 pm, Kaz Kylheku wrote: > On 2011-12-07, Andrew Tomazos wrote: > > > Summary: We want to find out how often a given token appears in a > > random stream formed by concatenating randomly chosen strings from a > > given set of strings. > > (Note hits can overlap each other) > > But tokens do not overlap, so you're not actually extracting tokens. Using > C tokens as an example, the C token >>= is one hit, not four. The longest > match calls for extracting three characters and moving on. Substitute occurrences of the word "token" in my post for "key string" (or just "string") and reinterpret. -Andrew. [I suppose, but finding tokens would be more interesting. -John]