Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!de-l.enfer-du-nord.net!feeder2.enfer-du-nord.net!news.swapon.de!fu-berlin.de!uni-berlin.de!individual.net!not-for-mail
From: Simon <bad@example.com>
Newsgroups: comp.lang.php
Subject: Tokenize an HTML page.
Date: Mon, 26 Nov 2012 09:56:46 +0200
Lines: 26
Message-ID: <ahgllvFmrgjU1@mid.individual.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Trace: individual.net bgJjKjoIuLjyXtyOzKNHfgzdxfZ0DgBGKB+E9LhHj0AMx4OVk7v10Jjy1k6bW3b3Vp
Cancel-Lock: sha1:2VcvMKWIgVfZwv5yEPhEQTxrB/g=
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Thunderbird/17.0
Xref: csiph.com comp.lang.php:7641

Hi,

I would like index a whole bunch of html documents on my site to speed 
up my internal searches, (I currently use 'LIKE "%...%"' and that's not 
very efficient).

My understanding would be to:
1) Remove some html (with strip_tags( ... ))
2) Walk the string and, every time I come across a stop character, 
(<space>,',",?,! etc...), then count that as a word.

The above solution is over simplistic as it does not work for many 
languages, (Hebrew for example uses the single quote as part of the word).

Also stripping HTML assumes that it is properly formated, something I 
cannot really guaranty, (and in any case, I might want to keep certain 
items such as websites inside the href='' tags).

So, before I re-invent the wheel, can someone suggest a 
script/class/code that is able to tokenize html content?

Any suggestions?

Many thanks

Simon