Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.javascript > #15757 > unrolled thread

regexp help

Back to article view | Back to comp.lang.javascript

Contents

  regexp help Andrew Poulos <ap_prog@hotmail.com> - 2012-09-07 14:37 +1000
    Re: regexp help Jake Jarvis <pig_in_shoes@yahoo.com> - 2012-09-07 10:55 +0200
      Re: regexp help Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2012-09-07 15:36 +0200
        Re: regexp help Andrew Poulos <ap_prog@hotmail.com> - 2012-09-08 10:27 +1000
          Re: regexp help Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2012-09-08 09:34 +0200
        Re: regexp help "Scott Sauyet" <3103invalid@webnews.fastusenet.org> - 2012-09-11 16:51 +0000
          Re: regexp help Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2012-09-11 22:31 +0200
            Re: regexp help "Scott Sauyet" <3103invalid@webnews.fastusenet.org> - 2012-09-19 11:56 +0000
              Re: regexp help Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2012-09-19 18:51 +0200
                Re: regexp help Scott Sauyet <scott.sauyet@gmail.com> - 2012-09-19 12:52 -0700
                  Re: regexp help Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2012-09-20 02:13 +0200
                    Re: regexp help Scott Sauyet <scott.sauyet@gmail.com> - 2012-09-20 06:51 -0700
                  Re: regexp help Dr J R Stockton <reply1237@merlyn.demon.co.uk.invalid> - 2012-09-20 20:23 +0100
        Re: regexp help "Michael Haufe (TNO)" <tno@thenewobjective.com> - 2012-09-11 14:55 -0700

#15757 — regexp help

I have a string
var str = 'This is some text.';

Within that string I need to find whether a whole word exists. The test 
is to be case insensitive. The word to test for is held in the var 'k'.

When I try this:

var k = 'SOME';
var myRe = new RegExp("b" + k + "b", "i");
var found = myRe.test(str);

I expected 'found' to return true. What am I doing wrong?

Andrew Poulos

[toc] | [next] | [standalone]

#15760

On 07.09.2012 06:37, wrote Andrew Poulos:
> I have a string
> var str = 'This is some text.';
> 
> Within that string I need to find whether a whole word exists. The test
> is to be case insensitive. The word to test for is held in the var 'k'.
> 
> When I try this:
> 
> var k = 'SOME';
> var myRe = new RegExp("b" + k + "b", "i");
> var found = myRe.test(str);
> 
> I expected 'found' to return true. What am I doing wrong?
> 

You are not escaping the two 'b' characters.

There are 2 levels here: the level of the regular expression in which
you want '\b', and the level of the pattern string in which you need
'\\b', because strings have their own escape rules which apply first and
under those '\b' means "backspace character", whereas '\\b' means
"backslash b".

-- 
Jake Jarvis

[toc] | [prev] | [next] | [standalone]

#15763

Jake Jarvis wrote:

> On 07.09.2012 06:37, wrote Andrew Poulos:
>> I have a string
>> var str = 'This is some text.';
>> 
>> Within that string I need to find whether a whole word exists. The test
>> is to be case insensitive. The word to test for is held in the var 'k'.
>> 
>> When I try this:
>> 
>> var k = 'SOME';
>> var myRe = new RegExp("b" + k + "b", "i");
>> var found = myRe.test(str);
>> 
>> I expected 'found' to return true. What am I doing wrong?
> 
> You are not escaping the two 'b' characters.
> 
> There are 2 levels here: the level of the regular expression in which
> you want '\b', and the level of the pattern string in which you need
> '\\b', because strings have their own escape rules which apply first and
> under those '\b' means "backspace character", whereas '\\b' means
> "backslash b".

In addition, \b is not sufficient in most cases as it matches a boundary 
between a "word character" and "not-a-word-character".  Word characters are 
those matched by `\w', which in turn is defined as the equivalent of [0-9A-
Za-z_].  Non-ASCII letters are not included.  Even basically English text 
can contain non-ASCII characters (borrowed from or written in other 
languages):

<http://stackoverflow.com/questions/1015844/how-to-remove-all-characters-
from-a-string/8568325#8568325> f.

Which gives me the idea that a Unicode mode for jsx.regexp.RegExp() to 
replace \b, \w and \W – in addition to or using the already implemented 
Unicode character properties (\p{Lu} et al.) – would come in handy.  
Comments anyone?


PointedEars
-- 
Sometimes, what you learn is wrong. If those wrong ideas are close to the 
root of the knowledge tree you build on a particular subject, pruning the 
bad branches can sometimes cause the whole tree to collapse.
  -- Mike Duffy in cljs, <news:Xns9FB6521286DB8invalidcom@94.75.214.39>

[toc] | [prev] | [next] | [standalone]

#15790

On 7/09/2012 11:36 PM, Thomas 'PointedEars' Lahn wrote:
> Jake Jarvis wrote:
>
>> On 07.09.2012 06:37, wrote Andrew Poulos:
>>> I have a string
>>> var str = 'This is some text.';
>>>
>>> Within that string I need to find whether a whole word exists. The test
>>> is to be case insensitive. The word to test for is held in the var 'k'.
>>>
>>> When I try this:
>>>
>>> var k = 'SOME';
>>> var myRe = new RegExp("b" + k + "b", "i");
>>> var found = myRe.test(str);
>>>
>>> I expected 'found' to return true. What am I doing wrong?
>>
>> You are not escaping the two 'b' characters.
>>
>> There are 2 levels here: the level of the regular expression in which
>> you want '\b', and the level of the pattern string in which you need
>> '\\b', because strings have their own escape rules which apply first and
>> under those '\b' means "backspace character", whereas '\\b' means
>> "backslash b".
>
> In addition, \b is not sufficient in most cases as it matches a boundary
> between a "word character" and "not-a-word-character".  Word characters are
> those matched by `\w', which in turn is defined as the equivalent of [0-9A-
> Za-z_].  Non-ASCII letters are not included.  Even basically English text
> can contain non-ASCII characters (borrowed from or written in other
> languages):

Hmm, that means that common apostrophe'd words such as "don't" or 
"Andrew's" would not be matched. BTW is "don't" considered a single word?

> <http://stackoverflow.com/questions/1015844/how-to-remove-all-characters-
> from-a-string/8568325#8568325> f.
>
> Which gives me the idea that a Unicode mode for jsx.regexp.RegExp() to
> replace \b, \w and \W – in addition to or using the already implemented
> Unicode character properties (\p{Lu} et al.) – would come in handy.
> Comments anyone?

I understand what you're suggesting but I don't know enough about the 
topic to make suggestions.

Andrew Poulos

[toc] | [prev] | [next] | [standalone]

#15791

Andrew Poulos wrote:

> On 7/09/2012 11:36 PM, Thomas 'PointedEars' Lahn wrote:
>> Jake Jarvis wrote:
>>> On 07.09.2012 06:37, wrote Andrew Poulos:
>>>> I have a string
>>>> var str = 'This is some text.';
>>>>
>>>> Within that string I need to find whether a whole word exists. The test
>>>> is to be case insensitive. The word to test for is held in the var 'k'.
>>>>
>>>> When I try this:
>>>>
>>>> var k = 'SOME';
>>>> var myRe = new RegExp("b" + k + "b", "i");
>>>> var found = myRe.test(str);
>>>>
>>>> I expected 'found' to return true. What am I doing wrong?
>>>
>>> You are not escaping the two 'b' characters.
>>>
>>> There are 2 levels here: the level of the regular expression in which
>>> you want '\b', and the level of the pattern string in which you need
>>> '\\b', because strings have their own escape rules which apply first and
>>> under those '\b' means "backspace character", whereas '\\b' means
>>> "backslash b".
>>
>> In addition, \b is not sufficient in most cases as it matches a boundary
>> between a "word character" and "not-a-word-character".  Word characters
>> are those matched by `\w', which in turn is defined as the equivalent of
>> [0-9A-Za-z_].  Non-ASCII letters are not included.  Even basically
>> English text can contain non-ASCII characters (borrowed from or written
>> in other languages):
> 
> Hmm, that means that common apostrophe'd words such as "don't" or
> "Andrew's" would not be matched.

Why don't you try?  They would:

  \b d o n \b ' \b t \b
   | | | |    |    |  |
  \b d o n    '    t \b

The problem is here, for example:

 \b B a h n \b ü \b b e r g a n g \b  ("railroad crossing" in German)
  | | | | |  |
 \b B a h n \b                        ("railroad" in German)

> BTW is "don't" considered a single word?

That depends on how you define a word.  With the native \w+: No, because it 
contains one character that is not a word character, it would be considered 
two words, and there would be two matches with /\w+/g.

>> <http://stackoverflow.com/questions/1015844/how-to-remove-all-characters-
>> from-a-string/8568325#8568325> f.
>>
>> Which gives me the idea that a Unicode mode for jsx.regexp.RegExp() to
>> replace \b, \w and \W – in addition to or using the already implemented
>> Unicode character properties (\p{Lu} et al.) – would come in handy.
>> Comments anyone?
> 
> I understand what you're suggesting but I don't know enough about the
> topic to make suggestions.

IOW, would you find it useful to match words containing non-ASCII letters 
with `new jsx.regexp.RegExp("\\w+", "u")'?


PointedEars
-- 
    realism:    HTML 4.01 Strict
    evangelism: XHTML 1.0 Strict
    madness:    XHTML 1.1 as application/xhtml+xml
                                                    -- Bjoern Hoehrmann

[toc] | [prev] | [next] | [standalone]

#15854

Thomas 'PointedEars' Lahn wrote:
> [ ... ]
> Which gives me the idea that a Unicode mode for jsx.regexp.RegExp() to
> replace \b, \w and \W � in addition to or using the already implemented
> Unicode character properties (\p{Lu} et al.) � would come in handy. �
> Comments anyone?

I think this would be very useful.  I haven't looked at the jsx.regexp implementation.  Would this be straightforward to add?

I guess I don't know the Unicode spec very well.  Are the characters that 
would reasonably be included inside an expanded `\w` mode within a few 
contiguous blocks?  Is the notion of what "would reasonably be included" 
even coherent?  Obviously you could make whatever call you choose about what 
to include, but is there an obvious call to make?

It's quite clear that one that included the extended Latin characters would 
be useful, and I imagine it would make sense for alphabets like Greek or 
Cyrillic, but the further afield we go from languages I know, the less I 
understand the implications.  Still, I would love to see it done, even if 
only for the simpler cases.

  -- Scott

[toc] | [prev] | [next] | [standalone]

#15860

Scott Sauyet wrote:

> Thomas 'PointedEars' Lahn wrote:
>> [ ... ]
>> Which gives me the idea that a Unicode mode for jsx.regexp.RegExp() to
>> replace \b, \w and \W � in addition to or using the already implemented
>> Unicode character properties (\p{Lu} et al.) � would come in handy. �
>> Comments anyone?
> 
> I think this would be very useful.

I figured as much :)  Thank you for your feedback.

> I haven't looked at the jsx.regexp implementation.  Would this be
> straightforward to add?

It was not straightforward to implement (character property class 
definitions L, M, N, and Word had to be added, and the character class 
normalizer had to be generalized – so that `[^a\Wb]' is equivalent to `(?:
[^ab]|[\p{Word}])' *before* escape sequence expansion).

But – as my Twitter followers are already aware – I did it today.  
Currently, \w and \W only; \b is a bit more tricky.

<http://PointedEars.de/scripts/test/regexp>
<http://PointedEars.de/websvn/log.php?repname=JSX&path=%2Ftrunk%2F&isdir=1>

> I guess I don't know the Unicode spec very well.  Are the characters that
> would reasonably be included inside an expanded `\w` mode within a few
> contiguous blocks?

No, but the character class can be built from existing character property 
class definitions, which more or less belong to contiguous blocks within the 
Unicode character set.

So far, I have implemented `\w' in Unicode mode to be equivalent to 
`[\p{L}\p{M}\p{N}\p{Pc}]'; that is, it includes letters, marks, numbers, and 
connector punctuation (like the underscore).  The unit test needs to be 
refined of course, but the results so far are promising.

> Is the notion of what "would reasonably be included" even coherent? 

Yes, the Unicode standard assigns properties to characters, such as Letter 
(L).  I had already created a version of the Unicode Character 
Database (UCD) in ECMAScript (UnicodeData.js), and a parser for the plain-
text version of the UCD (UnicodeData.txt, which can be freely downloaded 
from unicode.org), to replace escape sequences with character classes.

> Obviously you could make whatever call you choose about
> what to include, but is there an obvious call to make?

Yes, Perl already specifies it, so I followed perlrecharclass(1) (but I 
forgot to include \p{N}; that has been fixed in the meantime).

However, I can see now that it is wrong to call my implementation of \w in 
Unicode mode *PCRE*-conforming, as pcrepattern(1) specifies it differently: 
\w is equivalent to [\p{L}\p{N}_] there, while according to 
perlrecharclass(1) it appears to be [\p{L}\p{M}\p{N}\p{Pc}] in Perl.

I think that Perl's definition is better than PCRE's here as it also 
includes combining marks (M) like COMBINING TILDE, so that e. g. \w matches 
both "n" with COMBINING TILDE (common on Macs), and the non-combining letter 
"ñ" (more common on non-Macs), and more word-connecting punctuation than 
just the underscore.

But the property class value is no longer hardcoded in regexp.js.  So if you 
do not like it, you can either modify UnicodeData.js (while adhering to the 
terms of the license, of course), or call

  jsx.regexp.RegExp.definePropertyClasses({Word: "…"});

before you call jsx.regexp.RegExp().

> It's quite clear that one that included the extended Latin characters
> would be useful, and I imagine it would make sense for alphabets like
> Greek or Cyrillic, but the further afield we go from languages I know, the
> less I understand the implications.  Still, I would love to see it done,
> even if only for the simpler cases.

Please try regexp.js (in fact, please try all of JSX).  Any (constructive) 
feedback (here or elsewhere) is welcome.  (I am still working on anonymous 
SVN read access, and privileged SVN write access, before I will probably 
switch to git in the long term.)


Regards,

PointedEars
-- 
Anyone who slaps a 'this page is best viewed with Browser X' label on
a Web page appears to be yearning for the bad old days, before the Web,
when you had very little chance of reading a document written on another
computer, another word processor, or another network. -- Tim Berners-Lee

[toc] | [prev] | [next] | [standalone]

#15993

Thomas 'PointedEars' Lahn wrote:
> Scott Sauyet wrote:
>> Thomas 'PointedEars' Lahn wrote:
>>> [ ... ]
>>> Which gives me the idea that a Unicode mode for jsx.regexp.RegExp() to
>>> replace \b, \w and \W in addition to or using the already implemented
>>> Unicode character properties (\p{Lu} et al.) would come in handy.
> [ ... ]
>> I haven't looked at the jsx.regexp implementation. �Would this be
>> straightforward to add?
> 
> It was not straightforward to implement (character property class
> definitions L, M, N, and Word had to be added, and the character class
> normalizer had to be generalized � so that `[^a\Wb]' is equivalent to `(?:
> [^ab]|[\p{Word}])' *before* escape sequence expansion).
> 
> But � as my Twitter followers are already aware � I did it today. �
> Currently, \w and \W only; \b is a bit more tricky. 

Yes, I dug around in the implementation a bit this morning, only enough to see 
the broad outlines of what you're doing.  It looks interesting, and it looks 
challenging.  And it looks entirely worth doing.  I have no idea yet of how well 
you've managed to accomplish it.  So maybe I shouldn't be as impressed as I am.  
:-)

>> [ ... ]
>> Is the notion of what "would reasonably be included [in an expanded `\w`
>> mode]" even coherent?
> 
> Yes, the Unicode standard assigns properties to characters, such as Letter
> (L). �I had already created a version of the Unicode Character
> Database (UCD) in ECMAScript (UnicodeData.js), and a parser for the plain-
> text version of the UCD (UnicodeData.txt, which can be freely downloaded
> from unicode.org), to replace escape sequences with character classes.

Ok, that makes sense.  I'm clearly going to have to spend more time on this.


>> Obviously you could make whatever call you choose about
>> what to include, but is there an obvious call to make?
> 
> Yes, Perl already specifies it, so I followed perlrecharclass(1) (but I
> forgot to include \p{N}; that has been fixed in the meantime).
> 
> However, I can see now that it is wrong to call my implementation of \w in
> Unicode mode *PCRE*-conforming, as pcrepattern(1) specifies it differently:
> \w is equivalent to [\p{L}\p{N}_] there, while according to
> perlrecharclass(1) it appears to be [\p{L}\p{M}\p{N}\p{Pc}] in Perl.
> 
> I think that Perl's definition is better than PCRE's here [ ... ]

So yes, there are obvious choices, but which of the obvious choices is best is 
still up for grabs?  :-)

>> [ ... ]
> Please try regexp.js (in fact, please try all of JSX). �Any (constructive)
> feedback (here or elsewhere) is welcome. �(I am still working on anonymous
> SVN read access, and privileged SVN write access, before I will probably
> switch to git in the long term.)

I've been reading from the SVN web front end you supply, and haven't looked for 
plain downloads yet.

I mostly have used JSX the way I do several other libraries by regulars here, as 
sources of inspiration, sometimes looking to say, "Ah, so that's how Garrett 
chooses to do it, and that's how Thomas does it, and that's how David's works.  
I've never incorporated any of these libraries as they stand into projects.  I'm 
not quite sure why except that usually for my own projects, I like to roll my 
own code and for work, I'm almost always constrained to use certain libraries.  
(You can guess which ones!)

But I will try JSX regexp the next time I need to do something interesting with 
regex.

Thank you for your effort with this.

  -- Scott

[toc] | [prev] | [next] | [standalone]

#16002

Scott Sauyet wrote:

> Thomas 'PointedEars' Lahn wrote:
>> Scott Sauyet wrote:
>>> Thomas 'PointedEars' Lahn wrote:
>>>> [ ... ]
>>>> Which gives me the idea that a Unicode mode for jsx.regexp.RegExp() to
>>>> replace \b, \w and \W in addition to or using the already implemented
>>>> Unicode character properties (\p{Lu} et al.) would come in handy.
>> [ ... ]
>>> I haven't looked at the jsx.regexp implementation. �Would this be
>>> straightforward to add?
>> 
>> It was not straightforward to implement (character property class
>> definitions L, M, N, and Word had to be added, and the character class
>> normalizer had to be generalized � so that `[^a\Wb]' is equivalent to
>> `(?: [^ab]|[\p{Word}])' *before* escape sequence expansion).
>> 
>> But � as my Twitter followers are already aware � I did it today. �
>> Currently, \w and \W only; \b is a bit more tricky.

(You want to do something about that encoding problem of your posting 
agent.)
 
> Yes, I dug around in the implementation a bit this morning, only enough to
> see the broad outlines of what you're doing.  It looks interesting, and it
> looks challenging.  And it looks entirely worth doing.  I have no idea yet
> of how well you've managed to accomplish it.  So maybe I shouldn't be as
> impressed as I am. :-)

The JSdoc for jsx.regexp.RegExp has grown rather descriptive in the meantime 
(good that Resource Builder can automagically remove them for deployment) :)

If you re-run the unit test, you will notice that, based on the 
implementation for \w, it would appear that in the meantime I have mostly 
implemented \b in Unicode mode, and pattern-matching modifiers (?sx-sx) for 
the already implemented `s' (PCRE_DOTALL) and `x' (PCRE_EXTENDED) flags:

<http://PointedEars.de/scripts/test/regexp>

I am working on the rest of \b (`\b' after `('s and `\b' with match(/…/g) 
still need fixing), and other character class escapes, like \d and \s, in 
Unicode mode.  Although Perl and PCRE do not support it, given the current 
implementation of \w, \b, and \p{…}, it might be possible to implement in 
ECMAScript (?u-u) for `u' (would anyone find this useful?).  And then 
there's the greater challenge of implementing `(?im-im)'.
 
>>> Obviously you could make whatever call you choose about
>>> what to include, but is there an obvious call to make?
>> 
>> Yes, Perl already specifies it, so I followed perlrecharclass(1) (but I
>> forgot to include \p{N}; that has been fixed in the meantime).
>> 
>> However, I can see now that it is wrong to call my implementation of \w
>> in Unicode mode *PCRE*-conforming, as pcrepattern(1) specifies it
>> differently: \w is equivalent to [\p{L}\p{N}_] there, while according to
>> perlrecharclass(1) it appears to be [\p{L}\p{M}\p{N}\p{Pc}] in Perl.
>> 
>> I think that Perl's definition is better than PCRE's here [ ... ]
> 
> So yes, there are obvious choices, but which of the obvious choices is
> best is still up for grabs?  :-)
          ^^^^^^^^^^^^^^^^^^
I am not sure I fully understand that idiom.  However, saying that I think 
Perl's definition is better with \w, \W, and \b, already indicated that I 
intended to implement that instead of PCRE's.  So I did.  But AISB, if you 
are not happy with that, you can easily redefine the escape sequence, even 
during runtime (provided that the UCD has been loaded before; I am working 
on that as well):

  var RegExp2 = jsx.regexp.RegExp;

  /* \p{L}\p{M}\p{N}\p{Pc} */
  console.log(RegExp2.propertyClasses["Word"]);

  /* \w, \p{Word} à la PCRE */
  RegExp2.definePropertyClasses({
    Word: "\\p{L}\\p{N}_"
  });

  /* see above */
  var rx = new RegExp2("\\b(?<foo>\\w+\\b)", "u");

  var input = "-trärä-";

  var m = rx.exec(input);

  if (m)
  {
    /* "trärä" "trärä" "trärä"¹ */
    console.log(m[0], m[1], m.groups["foo"]);
  }

  /* trärä */
  console.log((new jsx.regexp.String(input)).match(rx).groups["foo"]);
 
> I mostly have used JSX the way I do several other libraries by regulars
> here, as sources of inspiration, sometimes looking to say, "Ah, so that's
> how Garrett chooses to do it, and that's how Thomas does it, and that's
> how David's works.  I've never incorporated any of these libraries as they
> stand into projects.  I'm not quite sure why except that usually for my
> own projects, I like to roll my own code and for work, I'm almost always
> constrained to use certain libraries. (You can guess which ones!)

ACK.

> But I will try JSX regexp the next time I need to do something interesting
> with regex.

Thanks.
 
> Thank you for your effort with this.

My pleasure.  That ECMAScript supports Unicode natively but TC39 – including 
Brendan Eich, unfortunately – cannot find it within themselves to add a 
Unicode mode for RegExp (let alone PCRE_EXTENDED and PCRE_DOTALL and other 
useful features found in other scripting languages) has bothered me for 
years.  Finally I find the time to do something about it.


PointedEars
___________
¹  No, it's _not_ a premature carnival joke ;-)
-- 
Anyone who slaps a 'this page is best viewed with Browser X' label on
a Web page appears to be yearning for the bad old days, before the Web,
when you had very little chance of reading a document written on another
computer, another word processor, or another network. -- Tim Berners-Lee

[toc] | [prev] | [next] | [standalone]

#16014

Thomas 'PointedEars' Lahn wrote:
> Scott Sauyet wrote:
>> Thomas 'PointedEars' Lahn wrote:
>>> Scott Sauyet wrote:
>>>> Thomas 'PointedEars' Lahn wrote:
>>>>> [ ... ]
>>>>> Which gives me the idea that a Unicode mode for jsx.regexp.RegExp() to
>>>>> replace \b, \w and \W in addition to or using the already implemented
>>>>> Unicode character properties (\p{Lu} et al.) would come in handy.
>>> [ ... ]
>>>> I haven't looked at the jsx.regexp implementation. Would this be
>>>> straightforward to add?
>
>>> It was not straightforward to implement (character property class
>>> definitions L, M, N, and Word had to be added, and the character class
>>> normalizer had to be generalized so that `[^a\Wb]' is equivalent to
>>> `(?: [^ab]|[\p{Word}])' *before* escape sequence expansion).
>
>>> But as my Twitter followers are already aware I did it today.
>>> Currently, \w and \W only; \b is a bit more tricky.
>
> (You want to do something about that encoding problem of your posting
> agent.)

Sorry, with how terrible Google Groups has gotten, I've looked around
for some other web-based solution I can use from work.  It looks like
another one is failing.  I'll look from home to see what the symptoms
are.  From within FastUsenet.org and Google Groups I'm not seeing
whatever issue you're noting.  It's back to Google for now, though.


> [ ... ]
> <http://PointedEars.de/scripts/test/regexp>
>
> I am working on the rest of \b (`\b' after `('s and `\b' with match(/…/g)
> still need fixing), and other character class escapes, like \d and \s, in
> Unicode mode.  Although Perl and PCRE do not support it, given the current
> implementation of \w, \b, and \p{…}, it might be possible to implement in
> ECMAScript (?u-u) for `u' (would anyone find this useful?).  And then
> there's the greater challenge of implementing `(?im-im)'.

I'm glad to hear you're making progress.  I wish I had something to
offer beyond encouragement, but at least you have my best wishes!


>>>> Obviously you could make whatever call you choose about
>>>> what to include, but is there an obvious call to make?
>
>>> Yes, Perl already specifies it, so I followed perlrecharclass(1) (but I
>>> forgot to include \p{N}; that has been fixed in the meantime).
>
>>> However, I can see now that it is wrong to call my implementation of \w
>>> in Unicode mode *PCRE*-conforming, as pcrepattern(1) specifies it
>>> differently: \w is equivalent to [\p{L}\p{N}_] there, while according to
>>> perlrecharclass(1) it appears to be [\p{L}\p{M}\p{N}\p{Pc}] in Perl.
>
>>> I think that Perl's definition is better than PCRE's here [ ... ]
>
>> So yes, there are obvious choices, but which of the obvious choices is
>> best is still up for grabs?  :-)
>
>           ^^^^^^^^^^^^^^^^^^
> I am not sure I fully understand that idiom.

I apologize.  Approximately, "unclaimed, available to anyone."

    <http://idioms.thefreedictionary.com/up+for+grabs>

>                                               However, saying that I think
> Perl's definition is better with \w, \W, and \b, already indicated that I
> intended to implement that instead of PCRE's.  So I did.  But AISB, if you
> are not happy with that, you can easily redefine the escape sequence, even
> during runtime (provided that the UCD has been loaded before; I am working
> on that as well):
>
>   var RegExp2 = jsx.regexp.RegExp;
>
>   /* \p{L}\p{M}\p{N}\p{Pc} */
>   console.log(RegExp2.propertyClasses["Word"]);
>
>   /* \w, \p{Word} à la PCRE */
>   RegExp2.definePropertyClasses({
>     Word: "\\p{L}\\p{N}_"
>   });
>
>   /* see above */
>   var rx = new RegExp2("\\b(?<foo>\\w+\\b)", "u");
>
>   var input = "-trärä-";
>
>   var m = rx.exec(input);
>
>   if (m)
>   {
>     /* "trärä" "trärä" "trärä"¹ */
>     console.log(m[0], m[1], m.groups["foo"]);
>   }
>
>   /* trärä */
>   console.log((new jsx.regexp.String(input)).match(rx).groups["foo"]);

That's an excellent approach, but I would suggest you think very hard
about your defaults, because that is likely how most users would use
it, regardless of how configurable you make it.

> [ ... ]
>> Thank you for your effort with this.
>
> My pleasure.  That ECMAScript supports Unicode natively but TC39 – including
> Brendan Eich, unfortunately – cannot find it within themselves to add a
> Unicode mode for RegExp (let alone PCRE_EXTENDED and PCRE_DOTALL and other
> useful features found in other scripting languages) has bothered me for
> years.  Finally I find the time to do something about it.

It's not something that's been tripping me up.  One of the few
benefits of living in a mostly monolingual society, I suppose,
especially one with the smallest alphabet around!  :-(

But I can clearly see the benefits, and can only hope that having a
well-written implementation might help spur the TC39 into rethinking
this.

Thanks again,

  -- Scott

[toc] | [prev] | [next] | [standalone]

#16026

Scott Sauyet wrote:

> Thomas 'PointedEars' Lahn wrote:
>> (You want to do something about that encoding problem of your posting
>> agent.)
> 
> Sorry, with how terrible Google Groups has gotten, I've looked around
> for some other web-based solution I can use from work.  It looks like
> another one is failing.  I'll look from home to see what the symptoms
> are.  From within FastUsenet.org and Google Groups I'm not seeing
> whatever issue you're noting.

webnews.fastusenet.org used UTF-8, but replaced U+2013 EN DASH in the quoted 
text with U+0096, a control character.

> It's back to Google for now, though.

I would rather live with borken en-dashes.

>> [ ... ]
>> <http://PointedEars.de/scripts/test/regexp>
>>
>> I am working on the rest of \b (`\b' after `('s and `\b' with match(/…/g)
>> still need fixing), and other character class escapes, like \d and \s, in
>> Unicode mode.  Although Perl and PCRE do not support it, given the
>> current implementation of \w, \b, and \p{…}, it might be possible to
>> implement in ECMAScript (?u-u) for `u' (would anyone find this useful?). 
>> And then there's the greater challenge of implementing `(?im-im)'.
> 
> I'm glad to hear you're making progress.  I wish I had something to
> offer beyond encouragement, but at least you have my best wishes!

Thanks.  `\b' after `('s and `\b' with match(/…/g) have been implemented 
now.  More in the unit test and changelog.

>>>>> Obviously you could make whatever call you choose about
>>>>> what to include, but is there an obvious call to make?
>>>>
>>>> Yes, Perl already specifies it, so I followed perlrecharclass(1) (but I
>>>> forgot to include \p{N}; that has been fixed in the meantime).
>>>>
>>>> However, I can see now that it is wrong to call my implementation of \w
>>>> in Unicode mode *PCRE*-conforming, as pcrepattern(1) specifies it
>>>> differently: \w is equivalent to [\p{L}\p{N}_] there, while according
>>>> to perlrecharclass(1) it appears to be [\p{L}\p{M}\p{N}\p{Pc}] in Perl.
>>>>
>>>> I think that Perl's definition is better than PCRE's here [ ... ]
>>>
>>> So yes, there are obvious choices, but which of the obvious choices is
>>> best is still up for grabs?  :-)
>>
>> ^^^^^^^^^^^^^^^^^^
>> I am not sure I fully understand that idiom.
> 
> I apologize.  Approximately, "unclaimed, available to anyone."
> 
>     <http://idioms.thefreedictionary.com/up+for+grabs>

If you are implementing syntactical and semantical extensions through string 
values as I do, you are basically free to implement whatever you like.  But 
if the new feature should be used as a stand-in for an existing feature, it 
is a good idea if syntax and result do not conflict with user expectations.  
That would be those that could be expected from the ECMAScript Language 
Specification, 5.1 Edition, and other scripting languages that implement the 
same feature, at this point.
 
>> However, saying that I think Perl's definition is better with \w, \W, and
>> \b, already indicated that I intended to implement that instead of
>> PCRE's.  So I did.  But AISB, if you are not happy with that, you can
>> easily redefine the escape sequence, even during runtime (provided that
>> the UCD has been loaded before; I am working on that as well):
>>
>> var RegExp2 = jsx.regexp.RegExp;
>>
>> /* \p{L}\p{M}\p{N}\p{Pc} */
>> console.log(RegExp2.propertyClasses["Word"]);
>>
>> /* \w, \p{Word} à la PCRE */
>> RegExp2.definePropertyClasses({
>> Word: "\\p{L}\\p{N}_"
>> });
>>
>> /* see above */
>> var rx = new RegExp2("\\b(?<foo>\\w+\\b)", "u");
>>
>> var input = "-trärä-";
>>
>> var m = rx.exec(input);
>>
>> if (m)
>> {
>> /* "trärä" "trärä" "trärä"¹ */
>> console.log(m[0], m[1], m.groups["foo"]);
>> }
>>
>> /* trärä */
>> console.log((new jsx.regexp.String(input)).match(rx).groups["foo"]);
> 
> That's an excellent approach, but I would suggest you think very hard
> about your defaults, because that is likely how most users would use
> it, regardless of how configurable you make it.

From top to bottom, and left to right:

I think that Perl's definitions for \w, \d and \s provide the best defaults 
for Unicode mode, as by contrast to PCRE's it includes the widest range of 
suitable Unicode characters.  For example, when you want to match word 
characters in Unicode, you should not be restricted to uncomposed characters 
– IMHO what matters is the final shape of the character and the meaning 
associated with that (such as, letter), not the code sequences that led to 
that shape.  With the PCRE definition apparently you would.

I think that jsx.regexp.RegExp can be used as a stand-in for RegExp() and 
RegExp literals is a good idea.  In fact, I am intentionally supporting 
references to RegExp instances as first argument as RegExp does: If you are 
not using syntactical extensions like named capturing groups, you can simply 
use jsx.regexp.RegExp(/…/gimy, "u") instead of /…/gimy to make your 
expression fully Unicode-aware (and thereby avoid the extra escaping that 
string values require).

Because it is defined so in PCRE, the named groups syntaxes can be found in 
Perl, PHP, and Python, among other programming languages that use the PCRE 
library.

I have chosen the `groups' property for matches for named capturing groups 
because it is so in Python (although the attribute it is a method there and 
accepts numbers as well); the same applies to other additional properties.

I also cannot see a viable alternative to jsx.regexp.String() if you need 
features that require syntactical extensions.  That is, `new 
jsx.regexp.String(string)' can be used as a stand-in for string values then; 
otherwise you can keep the primitive string value.

So I would like to think that I have thought very hard about my design and 
property class defaults already.  Do you have any specific issue with my 
approach, particularly with any of these considerations?


PointedEars
-- 
When all you know is jQuery, every problem looks $(olvable).

[toc] | [prev] | [next] | [standalone]

#16035

Thomas 'PointedEars' Lahn wrote:
> Scott Sauyet wrote:
>> Thomas 'PointedEars' Lahn wrote:
>>> (You want to do something about that encoding problem of your posting
>>> agent.)
>
>> Sorry, with how terrible Google Groups has gotten, I've looked around
>> for some other web-based solution I can use from work.  It looks like
>> another one is failing.  I'll look from home to see what the symptoms
>> are.  From within FastUsenet.org and Google Groups I'm not seeing
>> whatever issue you're noting.
>
> webnews.fastusenet.org used UTF-8, but replaced U+2013 EN DASH in the quoted
> text with U+0096, a control character.

That's sounds bizarre!

>> It's back to Google for now, though.
>
> I would rather live with borken en-dashes.

I'm afraid I may be without USENET at work unless I build some bridge
on my own website soon.  The current job seems to be blocking more and
more helpful web resources.  It's the first time I've started to chafe
at working for this particular large company.


>>> However, saying that I think Perl's definition is better with \w, \W, and
>>> \b, already indicated that I intended to implement that instead of
>>> PCRE's.  So I did.  But AISB, if you are not happy with that, you can
>>> easily redefine the escape sequence, even during runtime (provided that
>>> the UCD has been loaded before; I am working on that as well):

>>> [ ... ]
>> That's an excellent approach, but I would suggest you think very hard
>> about your defaults, because that is likely how most users would use
>> it, regardless of how configurable you make it.
>
> [ ... ]
> So I would like to think that I have thought very hard about my design and
> property class defaults already.  Do you have any specific issue with my
> approach, particularly with any of these considerations?

No, I'm afraid there's a little misunderstanding here.  I am not able
to invest the time to follow everything you're doing.  I like the
approach of choosing a sensible default but also making it
configurable.  I was applauding that, and noting that when APIs do so,
the default is almost always by far the most widely used.  So care
should be taken in choosing those defaults.  I did not mean to impugn
your choices or to suggest that you hadn't spent the time.  I just
have not been able to perform the analysis myself to determine if your
defaults seemed sensible, so I was suggesting (probably unnecessarily)
that you do so.  It looks as though you've considered this carefully
already.  I'm glad to hear it.

Again, thank you, and I look forward to being able to use this
implementation.

  -- Scott

[toc] | [prev] | [next] | [standalone]

#16062

In comp.lang.javascript message <86e0fb14-3158-435d-bbfd-540c093d5ed4@o8
g2000yqm.googlegroups.com>, Wed, 19 Sep 2012 12:52:24, Scott Sauyet
<scott.sauyet@gmail.com> posted:

>
>It's not something that's been tripping me up.  One of the few
>benefits of living in a mostly monolingual society, I suppose,
>especially one with the smallest alphabet around!  :-(

Not around here; Hangul has 24 letters.

And East Papuan Rotokas uses 12 letters, A E G I K O P R S T U V.

-- 
 (c) John Stockton, nr London UK               Reply address via Home Page.
   news:comp.lang.javascript FAQ <http://www.jibbering.com/faq/index.html>.
   <http://www.merlyn.demon.co.uk/js-index.htm> jscr maths, dates, sources.
   <http://www.merlyn.demon.co.uk/> TP/BP/Delphi/jscr/&c, FAQ items, links.

[toc] | [prev] | [next] | [standalone]

#15862

On Sep 7, 8:36 am, Thomas 'PointedEars' Lahn <PointedE...@web.de>
wrote:
> Jake Jarvis wrote:
> > On 07.09.2012 06:37, wrote Andrew Poulos:
> >> I have a string
> >> var str = 'This is some text.';
>
> >> Within that string I need to find whether a whole word exists. The test
> >> is to be case insensitive. The word to test for is held in the var 'k'.
>
> >> When I try this:
>
> >> var k = 'SOME';
> >> var myRe = new RegExp("b" + k + "b", "i");
> >> var found = myRe.test(str);
>
> >> I expected 'found' to return true. What am I doing wrong?
>
> > You are not escaping the two 'b' characters.
>
> > There are 2 levels here: the level of the regular expression in which
> > you want '\b', and the level of the pattern string in which you need
> > '\\b', because strings have their own escape rules which apply first and
> > under those '\b' means "backspace character", whereas '\\b' means
> > "backslash b".
>
> In addition, \b is not sufficient in most cases as it matches a boundary
> between a "word character" and "not-a-word-character".  Word characters are
> those matched by `\w', which in turn is defined as the equivalent of [0-9A-
> Za-z_].  Non-ASCII letters are not included.  Even basically English text
> can contain non-ASCII characters (borrowed from or written in other
> languages):
>
> <http://stackoverflow.com/questions/1015844/how-to-remove-all-characters-
> from-a-string/8568325#8568325> f.
>
> Which gives me the idea that a Unicode mode for jsx.regexp.RegExp() to
> replace \b, \w and \W – in addition to or using the already implemented
> Unicode character properties (\p{Lu} et al.) – would come in handy.
> Comments anyone?

Have you looked at the efforts of Steve Levithan's <http://xregexp.com/
> library? There may be some inspiration there.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.javascript

csiph-web