Re: Expected Token Density in Random Stream

From	Andrew Tomazos <andrew@tomazos.com>
Newsgroups	comp.compilers
Subject	Re: Expected Token Density in Random Stream
Date	2011-12-13 06:00 -0800
Organization	Compilers Central
Message-ID	<11-12-019@comp.compilers> (permalink)
References	<11-12-015@comp.compilers> <11-12-016@comp.compilers>

Show all headers | View raw

On Dec 11, 6:56 pm, Kaz Kylheku <k...@kylheku.com> wrote:
> On 2011-12-07, Andrew Tomazos <and...@tomazos.com> wrote:
>
> > Summary: We want to find out how often a given token appears in a
> > random stream formed by concatenating randomly chosen strings from a
> > given set of strings.
> > (Note hits can overlap each other)
>
> But tokens do not overlap, so you're not actually extracting tokens.  Using
> C tokens as an example, the C token >>= is one hit, not four.  The longest
> match calls for extracting three characters and moving on.

Substitute occurrences of the word "token" in my post for "key
string" (or just "string") and reinterpret.
  -Andrew.
[I suppose, but finding tokens would be more interesting. -John]

Back to comp.compilers | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Expected Token Density in Random Stream Andrew Tomazos <andrew@tomazos.com> - 2011-12-07 15:06 -0800
  Re: Expected Token Density in Random Stream Kaz Kylheku <kaz@kylheku.com> - 2011-12-11 17:56 +0000
    Re: Expected Token Density in Random Stream Andrew Tomazos <andrew@tomazos.com> - 2011-12-13 06:00 -0800
  Re: Expected Token Density in Random Stream Gene <gene.ressler@gmail.com> - 2011-12-19 17:48 -0800

csiph-web