Re: Expected Token Density in Random Stream

From	Kaz Kylheku <kaz@kylheku.com>
Newsgroups	comp.compilers
Subject	Re: Expected Token Density in Random Stream
Date	2011-12-11 17:56 +0000
Organization	A noiseless patient Spider
Message-ID	<11-12-016@comp.compilers> (permalink)
References	<11-12-015@comp.compilers>

Show all headers | View raw

On 2011-12-07, Andrew Tomazos <andrew@tomazos.com> wrote:
> Summary: We want to find out how often a given token appears in a
> random stream formed by concatenating randomly chosen strings from a
> given set of strings.

> (Note hits can overlap each other)

But tokens do not overlap, so you're not actually extracting tokens.  Using C
tokens as an example, the C token >>= is one hit, not four.  The longest match
calls for extracting three characters and moving on.

Back to comp.compilers | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Expected Token Density in Random Stream Andrew Tomazos <andrew@tomazos.com> - 2011-12-07 15:06 -0800
  Re: Expected Token Density in Random Stream Kaz Kylheku <kaz@kylheku.com> - 2011-12-11 17:56 +0000
    Re: Expected Token Density in Random Stream Andrew Tomazos <andrew@tomazos.com> - 2011-12-13 06:00 -0800
  Re: Expected Token Density in Random Stream Gene <gene.ressler@gmail.com> - 2011-12-19 17:48 -0800

csiph-web