Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!npeer.de.kpn-eurorings.net!npeer-ng0.de.kpn-eurorings.net!newsfeed.arcor.de!newsspool2.arcor-online.net!news.arcor.de.POSTED!not-for-mail Content-Type: text/plain; charset="UTF-8" Message-ID: <2835510.oF4MCyIXXA@PointedEars.de> From: Thomas 'PointedEars' Lahn Reply-To: Thomas 'PointedEars' Lahn Organization: PointedEars Software (PES) Date: Tue, 11 Sep 2012 22:31:20 +0200 User-Agent: KNode/4.4.11 Content-Transfer-Encoding: 8Bit X-Face: %i>XG-yXR'\"2P/C_aO%~;2o~?g0pPKmbOw^=NT`tprDEf++D.m7"}HW6.#=U:?2GGctkL,f89@H46O$ASoW&?s}.k+&. <1492793.DGNhQbDWfW@PointedEars.de> <201209111651133103@webnews.fastusenet.org> Followup-To: comp.lang.javascript MIME-Version: 1.0 Lines: 93 NNTP-Posting-Date: 11 Sep 2012 22:31:21 CEST NNTP-Posting-Host: be835dca.newsspool3.arcor-online.net X-Trace: DXC=T8MJ6T9U_NM@k=MdN::NBIMcF=Q^Z^V3H4Fo<]lROoRA8kF Thomas 'PointedEars' Lahn wrote: >> [ ... ] >> Which gives me the idea that a Unicode mode for jsx.regexp.RegExp() to >> replace \b, \w and \W � in addition to or using the already implemented >> Unicode character properties (\p{Lu} et al.) � would come in handy. � >> Comments anyone? > > I think this would be very useful. I figured as much :) Thank you for your feedback. > I haven't looked at the jsx.regexp implementation. Would this be > straightforward to add? It was not straightforward to implement (character property class definitions L, M, N, and Word had to be added, and the character class normalizer had to be generalized – so that `[^a\Wb]' is equivalent to `(?: [^ab]|[\p{Word}])' *before* escape sequence expansion). But – as my Twitter followers are already aware – I did it today. Currently, \w and \W only; \b is a bit more tricky. > I guess I don't know the Unicode spec very well. Are the characters that > would reasonably be included inside an expanded `\w` mode within a few > contiguous blocks? No, but the character class can be built from existing character property class definitions, which more or less belong to contiguous blocks within the Unicode character set. So far, I have implemented `\w' in Unicode mode to be equivalent to `[\p{L}\p{M}\p{N}\p{Pc}]'; that is, it includes letters, marks, numbers, and connector punctuation (like the underscore). The unit test needs to be refined of course, but the results so far are promising. > Is the notion of what "would reasonably be included" even coherent? Yes, the Unicode standard assigns properties to characters, such as Letter (L). I had already created a version of the Unicode Character Database (UCD) in ECMAScript (UnicodeData.js), and a parser for the plain- text version of the UCD (UnicodeData.txt, which can be freely downloaded from unicode.org), to replace escape sequences with character classes. > Obviously you could make whatever call you choose about > what to include, but is there an obvious call to make? Yes, Perl already specifies it, so I followed perlrecharclass(1) (but I forgot to include \p{N}; that has been fixed in the meantime). However, I can see now that it is wrong to call my implementation of \w in Unicode mode *PCRE*-conforming, as pcrepattern(1) specifies it differently: \w is equivalent to [\p{L}\p{N}_] there, while according to perlrecharclass(1) it appears to be [\p{L}\p{M}\p{N}\p{Pc}] in Perl. I think that Perl's definition is better than PCRE's here as it also includes combining marks (M) like COMBINING TILDE, so that e. g. \w matches both "n" with COMBINING TILDE (common on Macs), and the non-combining letter "ñ" (more common on non-Macs), and more word-connecting punctuation than just the underscore. But the property class value is no longer hardcoded in regexp.js. So if you do not like it, you can either modify UnicodeData.js (while adhering to the terms of the license, of course), or call jsx.regexp.RegExp.definePropertyClasses({Word: "…"}); before you call jsx.regexp.RegExp(). > It's quite clear that one that included the extended Latin characters > would be useful, and I imagine it would make sense for alphabets like > Greek or Cyrillic, but the further afield we go from languages I know, the > less I understand the implications. Still, I would love to see it done, > even if only for the simpler cases. Please try regexp.js (in fact, please try all of JSX). Any (constructive) feedback (here or elsewhere) is welcome. (I am still working on anonymous SVN read access, and privileged SVN write access, before I will probably switch to git in the long term.) Regards, PointedEars -- Anyone who slaps a 'this page is best viewed with Browser X' label on a Web page appears to be yearning for the bad old days, before the Web, when you had very little chance of reading a document written on another computer, another word processor, or another network. -- Tim Berners-Lee