Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.javascript > #15860

Re: regexp help

Followups directed to: comp.lang.javascript

Show all headers | View raw

Scott Sauyet wrote:

> Thomas 'PointedEars' Lahn wrote:
>> [ ... ]
>> Which gives me the idea that a Unicode mode for jsx.regexp.RegExp() to
>> replace \b, \w and \W � in addition to or using the already implemented
>> Unicode character properties (\p{Lu} et al.) � would come in handy. �
>> Comments anyone?
> 
> I think this would be very useful.

I figured as much :)  Thank you for your feedback.

> I haven't looked at the jsx.regexp implementation.  Would this be
> straightforward to add?

It was not straightforward to implement (character property class 
definitions L, M, N, and Word had to be added, and the character class 
normalizer had to be generalized – so that `[^a\Wb]' is equivalent to `(?:
[^ab]|[\p{Word}])' *before* escape sequence expansion).

But – as my Twitter followers are already aware – I did it today.  
Currently, \w and \W only; \b is a bit more tricky.

<http://PointedEars.de/scripts/test/regexp>
<http://PointedEars.de/websvn/log.php?repname=JSX&path=%2Ftrunk%2F&isdir=1>

> I guess I don't know the Unicode spec very well.  Are the characters that
> would reasonably be included inside an expanded `\w` mode within a few
> contiguous blocks?

No, but the character class can be built from existing character property 
class definitions, which more or less belong to contiguous blocks within the 
Unicode character set.

So far, I have implemented `\w' in Unicode mode to be equivalent to 
`[\p{L}\p{M}\p{N}\p{Pc}]'; that is, it includes letters, marks, numbers, and 
connector punctuation (like the underscore).  The unit test needs to be 
refined of course, but the results so far are promising.

> Is the notion of what "would reasonably be included" even coherent? 

Yes, the Unicode standard assigns properties to characters, such as Letter 
(L).  I had already created a version of the Unicode Character 
Database (UCD) in ECMAScript (UnicodeData.js), and a parser for the plain-
text version of the UCD (UnicodeData.txt, which can be freely downloaded 
from unicode.org), to replace escape sequences with character classes.

> Obviously you could make whatever call you choose about
> what to include, but is there an obvious call to make?

Yes, Perl already specifies it, so I followed perlrecharclass(1) (but I 
forgot to include \p{N}; that has been fixed in the meantime).

However, I can see now that it is wrong to call my implementation of \w in 
Unicode mode *PCRE*-conforming, as pcrepattern(1) specifies it differently: 
\w is equivalent to [\p{L}\p{N}_] there, while according to 
perlrecharclass(1) it appears to be [\p{L}\p{M}\p{N}\p{Pc}] in Perl.

I think that Perl's definition is better than PCRE's here as it also 
includes combining marks (M) like COMBINING TILDE, so that e. g. \w matches 
both "n" with COMBINING TILDE (common on Macs), and the non-combining letter 
"ñ" (more common on non-Macs), and more word-connecting punctuation than 
just the underscore.

But the property class value is no longer hardcoded in regexp.js.  So if you 
do not like it, you can either modify UnicodeData.js (while adhering to the 
terms of the license, of course), or call

  jsx.regexp.RegExp.definePropertyClasses({Word: "…"});

before you call jsx.regexp.RegExp().

> It's quite clear that one that included the extended Latin characters
> would be useful, and I imagine it would make sense for alphabets like
> Greek or Cyrillic, but the further afield we go from languages I know, the
> less I understand the implications.  Still, I would love to see it done,
> even if only for the simpler cases.

Please try regexp.js (in fact, please try all of JSX).  Any (constructive) 
feedback (here or elsewhere) is welcome.  (I am still working on anonymous 
SVN read access, and privileged SVN write access, before I will probably 
switch to git in the long term.)


Regards,

PointedEars
-- 
Anyone who slaps a 'this page is best viewed with Browser X' label on
a Web page appears to be yearning for the bad old days, before the Web,
when you had very little chance of reading a document written on another
computer, another word processor, or another network. -- Tim Berners-Lee

Back to comp.lang.javascript | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

regexp help Andrew Poulos <ap_prog@hotmail.com> - 2012-09-07 14:37 +1000
  Re: regexp help Jake Jarvis <pig_in_shoes@yahoo.com> - 2012-09-07 10:55 +0200
    Re: regexp help Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2012-09-07 15:36 +0200
      Re: regexp help Andrew Poulos <ap_prog@hotmail.com> - 2012-09-08 10:27 +1000
        Re: regexp help Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2012-09-08 09:34 +0200
      Re: regexp help "Scott Sauyet" <3103invalid@webnews.fastusenet.org> - 2012-09-11 16:51 +0000
        Re: regexp help Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2012-09-11 22:31 +0200
          Re: regexp help "Scott Sauyet" <3103invalid@webnews.fastusenet.org> - 2012-09-19 11:56 +0000
            Re: regexp help Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2012-09-19 18:51 +0200
              Re: regexp help Scott Sauyet <scott.sauyet@gmail.com> - 2012-09-19 12:52 -0700
                Re: regexp help Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2012-09-20 02:13 +0200
                Re: regexp help Scott Sauyet <scott.sauyet@gmail.com> - 2012-09-20 06:51 -0700
                Re: regexp help Dr J R Stockton <reply1237@merlyn.demon.co.uk.invalid> - 2012-09-20 20:23 +0100
      Re: regexp help "Michael Haufe (TNO)" <tno@thenewobjective.com> - 2012-09-11 14:55 -0700

csiph-web