Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!xlned.com!feeder7.xlned.com!news.astraweb.com!border5.a.newsrouter.astraweb.com!border2.nntp.ams.giganews.com!Xl.tags.giganews.com!border1.nntp.ams.giganews.com!nntp.giganews.com!local2.nntp.ams.giganews.com!nntp.bt.com!news.bt.com.POSTED!not-for-mail
NNTP-Posting-Date: Mon, 26 May 2014 06:33:27 -0500
From: "Chris Uppal" <chris.uppal@metagnostic.REMOVE-THIS.org>
Newsgroups: comp.lang.java.programmer
References: <375vn9th0bc1faa04eg136be53hbvfjq4v@4ax.com> <bubj58Fn42U1@mid.individual.net> <8m54o913tjf1sagk75ktn7a0960nugpovv@4ax.com> <lltcuf$3mn$1@dont-email.me>
Subject: Re: regex tidy
Date: Mon, 26 May 2014 12:31:17 +0100
X-Priority: 3
X-MSMail-Priority: Normal
X-Newsreader: Microsoft Outlook Express 6.00.2900.5512
X-RFC2646: Format=Flowed; Response
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5512
Message-ID: <FfGdnX6yK_gaux7OnZ2dnUVZ8iGdnZ2d@bt.com>
Lines: 25
X-Usenet-Provider: http://www.giganews.com
X-AuthenticatedUsername: NoAuthUser
X-Trace: sv3-LbQAv7zdni+DCk0U9xQqumSSKmY1Pw/5Eybc4duPDJ7hxeJRgbrcmwc53cvyJsmMiz+s0gTzBcuTyIb!79eJOERklTuzX0QUSyLNW0zZzzMRCLxJ7gqYIyELhwjojh5Bo0A2evZ0Qo7Hej97NFsRsuUsdA8=
X-Complaints-To: abuse@btinternet.com
X-DMCA-Complaints-To: abuse@btinternet.com
X-Abuse-and-DMCA-Info: Please be sure to forward a copy of ALL headers
X-Abuse-and-DMCA-Info: Otherwise we will be unable to process your complaint properly
X-Postfilter: 1.3.40
X-Original-Bytes: 2409
Xref: csiph.com comp.lang.java.programmer:28566

Joshua Cranmer ?? wrote:

> [Although this does raise the question of how regular expressions handle
> or fail to handle characters like ?? or the penguin in my display name.
> Stupid UTF-16 and the "it's fixed-width characters if you look at it
> funny" results.]

I don't know how the Java regexp implementation works (very badly, last time I 
looked, but that was /years/ ago) but you can easily handle variable-width 
characters by building the regexp over the underlying "words".

E.g. I (in another language) match UTF-8 patterns by transforming them into a 
FSA over the underlying byte sequences.  That increases the number of 
transitions needed to find a match to  be linear in the number of bytes of 
input rather than the number of characters, but means that my jump tables only 
have to handle 256 possible values instead of several million.

The test would be, does the trivial range "^["\uD8eD\uDC27"]$" (= 
^[<penguin>]$) match the input string "\uD8eD\uDC27" ("<penguin>"), or does it 
match either of the two input strings, "\uD8eD" or "\uDC27"?  (My UTF-16 
encodings may be wrong here, but you get the idea).

    -- chris