Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #35801 > unrolled thread

re: ignore case only for a part of the regex?

Back to article view | Back to comp.lang.python

Contents

  re: ignore case only for a part of the regex? Helmut Jarausch <jarausch@skynet.be> - 2012-12-30 11:24 +0000
    Re: ignore case only for a part of the regex? Roy Smith <roy@panix.com> - 2012-12-30 10:20 -0500
      Re: ignore case only for a part of the regex? Joel Goldstick <joel.goldstick@gmail.com> - 2012-12-30 10:41 -0500
      Re: ignore case only for a part of the regex? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-01-01 04:14 +0000
        Re: ignore case only for a part of the regex? Vlastimil Brom <vlastimil.brom@gmail.com> - 2013-01-02 00:09 +0100
          Re: ignore case only for a part of the regex? wxjmfauth@gmail.com - 2013-01-02 06:17 -0800
          Re: ignore case only for a part of the regex? wxjmfauth@gmail.com - 2013-01-02 06:17 -0800
    Re: ignore case only for a part of the regex? Vlastimil Brom <vlastimil.brom@gmail.com> - 2012-12-30 17:38 +0100
      Re: ignore case only for a part of the regex? Roy Smith <roy@panix.com> - 2012-12-30 12:32 -0500
        Re: ignore case only for a part of the regex? MRAB <python@mrabarnett.plus.com> - 2012-12-30 18:04 +0000
        Re: ignore case only for a part of the regex? Cameron Simpson <cs@zip.com.au> - 2012-12-31 11:29 +1100

#35801 — re: ignore case only for a part of the regex?

Hi,

is there a means to specify that 'ignore-case' should only apply to a part
of a regex?

E.g.

the regex should match  Msg-id:, Msg-Id, ...  but not  msg-id: and so on.

I've tried the pattern
r'^Msg-(?:(?i)id):'
but (?i) makes the whole pattern ignoring case.

In my simple case I could say
r'Msg-[Ii][Dd]:'
but that's a bit clumsy.

Is there a more elegant way? Is there a way to compose a pattern
from subpatterns which are compiled with different flags?

Many thanks for a hint,
Helmut.

[toc] | [next] | [standalone]

#35803

Helmut Jarausch <jarausch@skynet.be> wrote:

> is there a means to specify that 'ignore-case' should only apply to a part
> of a regex?

Not that I'm aware of.
 
> the regex should match  Msg-id:, Msg-Id, ...  but not  msg-id: and so on.

What's the use-case for this?

The way I would typically do something like this is build my regexes in 
all lower case and .lower() the text I was matching against them.  I'm 
curious what you're doing where you want to enforce case sensitivity in 
one part of a header, but not in another.

[toc] | [prev] | [next] | [standalone]

#35804

[Multipart message — attachments visible in raw view] — view raw

On Sun, Dec 30, 2012 at 10:20 AM, Roy Smith <roy@panix.com> wrote:

> Helmut Jarausch <jarausch@skynet.be> wrote:
>
> > is there a means to specify that 'ignore-case' should only apply to a
> part
> > of a regex?
>

Python has excellent string methods.  There seems to be a split between
people who first always grab regex for string parsing, and those who  might
not.  If you go with your regex, I think you can comment what you have and
move on.  I glaze over looking at regexes.  That's just me. The code to
first search for "Msg-", then check what follows would take a couple of
lines, but might be easier to understand later.  I've been writing python
for a couple of years, and although I feel comfortable with it, there is
much more more me to learn.  One thing I have learned over many years of
programming is that figuring out what a piece of code is trying to
accomplish takes more time than writing it originally.

Do you really want to match "Msg-iD" (lower case i)? Or are you only
allowing "ID" or "Id"?

>
> Not that I'm aware of.
>
> > the regex should match  Msg-id:, Msg-Id, ...  but not  msg-id: and so on.
>
> What's the use-case for this?
>
> The way I would typically do something like this is build my regexes in
> all lower case and .lower() the text I was matching against them.  I'm
> curious what you're doing where you want to enforce case sensitivity in
> one part of a header, but not in another.
> --
> http://mail.python.org/mailman/listinfo/python-list
>



-- 
Joel Goldstick

[toc] | [prev] | [next] | [standalone]

#35876

On Sun, 30 Dec 2012 10:20:19 -0500, Roy Smith wrote:

> The way I would typically do something like this is build my regexes in
> all lower case and .lower() the text I was matching against them.  I'm
> curious what you're doing where you want to enforce case sensitivity in
> one part of a header, but not in another.

Well, sometimes you have things that are case sensitive, and other things 
which are not, and sometimes you need to match them at the same time. I 
don't think this is any more unusual than (say) wanting to match an 
otherwise lowercase word whether or not it comes at the start of a 
sentence:

"[Pp]rogramming"

is conceptually equivalent to "match case-insensitive `p`, and case-
sensitive `rogramming`".


By the way, although there is probably nothing you can (easily) do about 
this prior to Python 3.3, converting to lowercase is not the right way to 
do case-insensitive matching. It happens to work correctly for ASCII, but 
it is not correct for all alphabetic characters.


py> 'Straße'.lower()
'straße'
py> 'Straße'.upper()
'STRASSE'


The right way is to casefold first, then match:

py> 'Straße'.casefold()
'strasse'


Curiously, there is an uppercase ß in old German. In recent years some 
typographers have started using it instead of SS, but it's still rare, 
and the official German rules have ß transform into SS and vice versa. 
It's in Unicode, but few fonts show it:

py> unicodedata.lookup('LATIN CAPITAL LETTER SHARP S')
'ẞ'



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#35928

2013/1/1 Steven D'Aprano <steve+comp.lang.python@pearwood.info>:
> On Sun, 30 Dec 2012 10:20:19 -0500, Roy Smith wrote:
>
>> The way I would typically do something like this is build my regexes in
>> all lower case and .lower() the text I was matching against them.  I'm
>> curious what you're doing where you want to enforce case sensitivity in
>> one part of a header, but not in another.
>
> Well, sometimes you have things that are case sensitive, and other things
> which are not, and sometimes you need to match them at the same time. I
> don't think this is any more unusual than (say) wanting to match an
> otherwise lowercase word whether or not it comes at the start of a
> sentence:
>
> "[Pp]rogramming"
>
> is conceptually equivalent to "match case-insensitive `p`, and case-
> sensitive `rogramming`".
>
>
> By the way, although there is probably nothing you can (easily) do about
> this prior to Python 3.3, converting to lowercase is not the right way to
> do case-insensitive matching. It happens to work correctly for ASCII, but
> it is not correct for all alphabetic characters.
>
>
> py> 'Straße'.lower()
> 'straße'
> py> 'Straße'.upper()
> 'STRASSE'
>
>
> The right way is to casefold first, then match:
>
> py> 'Straße'.casefold()
> 'strasse'
>
>
> Curiously, there is an uppercase ß in old German. In recent years some
> typographers have started using it instead of SS, but it's still rare,
> and the official German rules have ß transform into SS and vice versa.
> It's in Unicode, but few fonts show it:
>
> py> unicodedata.lookup('LATIN CAPITAL LETTER SHARP S')
> 'ẞ'
>
>
>
> --
> Steven
> --
> http://mail.python.org/mailman/listinfo/python-list

Hi,
just for completeness, the mentioned regex library can take care of
casfolding in case insensitive matching (in all supported versions:
Python 2.5-2.7 and 3.1-3.3); i.e.:
# case sensitive match:
>>> for m in regex.findall(ur"Straße", u" STRAßE STRASSE STRAẞE Strasse Straße "): print m
...
Straße

# case insensitive match:
>>> for m in regex.findall(ur"(?i)Straße", u" STRAßE STRASSE STRAẞE Strasse Straße "): print m
...
STRAßE
STRAẞE
Straße

# case insensitive match with casefolding:
>>> for m in regex.findall(ur"(?if)Straße", u" STRAßE STRASSE STRAẞE Strasse Straße "): print m
...
STRAßE
STRASSE
STRAẞE
Strasse
Straße
>>>
>>>

# after enabling the backwards incompatible modern matching behaviour,
casefolding is by default turned on for case insensitive matches
>>> for m in regex.findall(ur"(?V1i)Straße", u" STRAßE STRASSE STRAẞE Strasse Straße "): print m
...
STRAßE
STRASSE
STRAẞE
Strasse
Straße
>>>


As a small addition, the originally posted pattern r'^Msg-(?:(?i)id):'
would actually work as expected in this modern matching mode in regex
- enabled with the V1 flag. In this case the flag-setting (?i) only
affects the following parts of the pattern, not the whole pattern like
in the current "re" and V0-compatibility-mode "regex"

>>> regex.findall(r"(?V1)Msg-(?:(?i)id):", "the regex should match  Msg-id:, Msg-Id:, ...  but not  msg-id:, MSG-ID:  and so on")
['Msg-id:', 'Msg-Id:']
>>>

regards,
      vbr

[toc] | [prev] | [next] | [standalone]

#35986

Le mercredi 2 janvier 2013 00:09:45 UTC+1, Vlastimil Brom a écrit :
> 2013/1/1 Steven D'Aprano <steve+comp.lang.python@pearwood.info>:
> 
> > On Sun, 30 Dec 2012 10:20:19 -0500, Roy Smith wrote:
> 
> >
> 
> >> The way I would typically do something like this is build my regexes in
> 
> >> all lower case and .lower() the text I was matching against them.  I'm
> 
> >> curious what you're doing where you want to enforce case sensitivity in
> 
> >> one part of a header, but not in another.
> 
> >
> 
> > Well, sometimes you have things that are case sensitive, and other things
> 
> > which are not, and sometimes you need to match them at the same time. I
> 
> > don't think this is any more unusual than (say) wanting to match an
> 
> > otherwise lowercase word whether or not it comes at the start of a
> 
> > sentence:
> 
> >
> 
> > "[Pp]rogramming"
> 
> >
> 
> > is conceptually equivalent to "match case-insensitive `p`, and case-
> 
> > sensitive `rogramming`".
> 
> >
> 
> >
> 
> > By the way, although there is probably nothing you can (easily) do about
> 
> > this prior to Python 3.3, converting to lowercase is not the right way to
> 
> > do case-insensitive matching. It happens to work correctly for ASCII, but
> 
> > it is not correct for all alphabetic characters.
> 
> >
> 
> >
> 
> > py> 'Straße'.lower()
> 
> > 'straße'
> 
> > py> 'Straße'.upper()
> 
> > 'STRASSE'
> 
> >
> 
> >
> 
> > The right way is to casefold first, then match:
> 
> >
> 
> > py> 'Straße'.casefold()
> 
> > 'strasse'
> 
> >
> 
> >
> 
> > Curiously, there is an uppercase ß in old German. In recent years some
> 
> > typographers have started using it instead of SS, but it's still rare,
> 
> > and the official German rules have ß transform into SS and vice versa.
> 
> > It's in Unicode, but few fonts show it:
> 
> >
> 
> > py> unicodedata.lookup('LATIN CAPITAL LETTER SHARP S')
> 
> > 'ẞ'
> 
> >
> 
> >
> 
> >
> 
> > --
> 
> > Steven
> 
> > --
> 
> > http://mail.python.org/mailman/listinfo/python-list
> 
> 
> 
> Hi,
> 
> just for completeness, the mentioned regex library can take care of
> 
> casfolding in case insensitive matching (in all supported versions:
> 
> Python 2.5-2.7 and 3.1-3.3); i.e.:
> 
> # case sensitive match:
> 
> >>> for m in regex.findall(ur"Straße", u" STRAßE STRASSE STRAẞE Strasse Straße "): print m
> 
> ...
> 
> Straße
> 
> 
> 
> # case insensitive match:
> 
> >>> for m in regex.findall(ur"(?i)Straße", u" STRAßE STRASSE STRAẞE Strasse Straße "): print m
> 
> ...
> 
> STRAßE
> 
> STRAẞE
> 
> Straße
> 
> 
> 
> # case insensitive match with casefolding:
> 
> >>> for m in regex.findall(ur"(?if)Straße", u" STRAßE STRASSE STRAẞE Strasse Straße "): print m
> 
> ...
> 
> STRAßE
> 
> STRASSE
> 
> STRAẞE
> 
> Strasse
> 
> Straße
> 
> >>>
> 
> >>>
> 
> 
> 
> # after enabling the backwards incompatible modern matching behaviour,
> 
> casefolding is by default turned on for case insensitive matches
> 
> >>> for m in regex.findall(ur"(?V1i)Straße", u" STRAßE STRASSE STRAẞE Strasse Straße "): print m
> 
> ...
> 
> STRAßE
> 
> STRASSE
> 
> STRAẞE
> 
> Strasse
> 
> Straße
> 
> >>>
> 
> 
> 
> 
> 
> As a small addition, the originally posted pattern r'^Msg-(?:(?i)id):'
> 
> would actually work as expected in this modern matching mode in regex
> 
> - enabled with the V1 flag. In this case the flag-setting (?i) only
> 
> affects the following parts of the pattern, not the whole pattern like
> 
> in the current "re" and V0-compatibility-mode "regex"
> 
> 
> 
> >>> regex.findall(r"(?V1)Msg-(?:(?i)id):", "the regex should match  Msg-id:, Msg-Id:, ...  but not  msg-id:, MSG-ID:  and so on")
> 
> ['Msg-id:', 'Msg-Id:']
> 

------

Vlastimil:

Excellent.

-----

Steven:

..." It's in Unicode, but few fonts show it:" ...

Das grosse Eszett is a member of the unicode subsets MES-2, WGL-4.
Good - serious - fonts are via OpenType MES-2 or WGL-4 compliant.
So, it is a no problem.

I do not know (and I did not check) if the code point, 1e9e, is part of
the utf32 table.

jmf

[toc] | [prev] | [next] | [standalone]

#35987

Le mercredi 2 janvier 2013 00:09:45 UTC+1, Vlastimil Brom a écrit :
> 2013/1/1 Steven D'Aprano <steve+comp.lang.python@pearwood.info>:
> 
> > On Sun, 30 Dec 2012 10:20:19 -0500, Roy Smith wrote:
> 
> >
> 
> >> The way I would typically do something like this is build my regexes in
> 
> >> all lower case and .lower() the text I was matching against them.  I'm
> 
> >> curious what you're doing where you want to enforce case sensitivity in
> 
> >> one part of a header, but not in another.
> 
> >
> 
> > Well, sometimes you have things that are case sensitive, and other things
> 
> > which are not, and sometimes you need to match them at the same time. I
> 
> > don't think this is any more unusual than (say) wanting to match an
> 
> > otherwise lowercase word whether or not it comes at the start of a
> 
> > sentence:
> 
> >
> 
> > "[Pp]rogramming"
> 
> >
> 
> > is conceptually equivalent to "match case-insensitive `p`, and case-
> 
> > sensitive `rogramming`".
> 
> >
> 
> >
> 
> > By the way, although there is probably nothing you can (easily) do about
> 
> > this prior to Python 3.3, converting to lowercase is not the right way to
> 
> > do case-insensitive matching. It happens to work correctly for ASCII, but
> 
> > it is not correct for all alphabetic characters.
> 
> >
> 
> >
> 
> > py> 'Straße'.lower()
> 
> > 'straße'
> 
> > py> 'Straße'.upper()
> 
> > 'STRASSE'
> 
> >
> 
> >
> 
> > The right way is to casefold first, then match:
> 
> >
> 
> > py> 'Straße'.casefold()
> 
> > 'strasse'
> 
> >
> 
> >
> 
> > Curiously, there is an uppercase ß in old German. In recent years some
> 
> > typographers have started using it instead of SS, but it's still rare,
> 
> > and the official German rules have ß transform into SS and vice versa.
> 
> > It's in Unicode, but few fonts show it:
> 
> >
> 
> > py> unicodedata.lookup('LATIN CAPITAL LETTER SHARP S')
> 
> > 'ẞ'
> 
> >
> 
> >
> 
> >
> 
> > --
> 
> > Steven
> 
> > --
> 
> > http://mail.python.org/mailman/listinfo/python-list
> 
> 
> 
> Hi,
> 
> just for completeness, the mentioned regex library can take care of
> 
> casfolding in case insensitive matching (in all supported versions:
> 
> Python 2.5-2.7 and 3.1-3.3); i.e.:
> 
> # case sensitive match:
> 
> >>> for m in regex.findall(ur"Straße", u" STRAßE STRASSE STRAẞE Strasse Straße "): print m
> 
> ...
> 
> Straße
> 
> 
> 
> # case insensitive match:
> 
> >>> for m in regex.findall(ur"(?i)Straße", u" STRAßE STRASSE STRAẞE Strasse Straße "): print m
> 
> ...
> 
> STRAßE
> 
> STRAẞE
> 
> Straße
> 
> 
> 
> # case insensitive match with casefolding:
> 
> >>> for m in regex.findall(ur"(?if)Straße", u" STRAßE STRASSE STRAẞE Strasse Straße "): print m
> 
> ...
> 
> STRAßE
> 
> STRASSE
> 
> STRAẞE
> 
> Strasse
> 
> Straße
> 
> >>>
> 
> >>>
> 
> 
> 
> # after enabling the backwards incompatible modern matching behaviour,
> 
> casefolding is by default turned on for case insensitive matches
> 
> >>> for m in regex.findall(ur"(?V1i)Straße", u" STRAßE STRASSE STRAẞE Strasse Straße "): print m
> 
> ...
> 
> STRAßE
> 
> STRASSE
> 
> STRAẞE
> 
> Strasse
> 
> Straße
> 
> >>>
> 
> 
> 
> 
> 
> As a small addition, the originally posted pattern r'^Msg-(?:(?i)id):'
> 
> would actually work as expected in this modern matching mode in regex
> 
> - enabled with the V1 flag. In this case the flag-setting (?i) only
> 
> affects the following parts of the pattern, not the whole pattern like
> 
> in the current "re" and V0-compatibility-mode "regex"
> 
> 
> 
> >>> regex.findall(r"(?V1)Msg-(?:(?i)id):", "the regex should match  Msg-id:, Msg-Id:, ...  but not  msg-id:, MSG-ID:  and so on")
> 
> ['Msg-id:', 'Msg-Id:']
> 

------

Vlastimil:

Excellent.

-----

Steven:

..." It's in Unicode, but few fonts show it:" ...

Das grosse Eszett is a member of the unicode subsets MES-2, WGL-4.
Good - serious - fonts are via OpenType MES-2 or WGL-4 compliant.
So, it is a no problem.

I do not know (and I did not check) if the code point, 1e9e, is part of
the utf32 table.

jmf

[toc] | [prev] | [next] | [standalone]

#35806

2012/12/30 Helmut Jarausch <jarausch@skynet.be>:
> Hi,
>
> is there a means to specify that 'ignore-case' should only apply to a part
> of a regex?
>
> E.g.
>
> the regex should match  Msg-id:, Msg-Id, ...  but not  msg-id: and so on.
>
> I've tried the pattern
> r'^Msg-(?:(?i)id):'
> but (?i) makes the whole pattern ignoring case.
>
> In my simple case I could say
> r'Msg-[Ii][Dd]:'
> but that's a bit clumsy.
>
> Is there a more elegant way? Is there a way to compose a pattern
> from subpatterns which are compiled with different flags?
>
> Many thanks for a hint,
> Helmut.
> --
> http://mail.python.org/mailman/listinfo/python-list

Hi,
you may check the new regex implementation for python
http://pypi.python.org/pypi/regex
which allows (among many other improvements) for scoped flags:

>>> import regex
>>> regex.findall(r"Msg-(?i:id):", "the regex should match  Msg-id:, Msg-Id:, ...  but not  msg-id:, MSG-ID:  and so on")
['Msg-id:', 'Msg-Id:']
>>>

hth,
 vbr

[toc] | [prev] | [next] | [standalone]

#35807

In article <mailman.1467.1356885520.29569.python-list@python.org>,
 Vlastimil Brom <vlastimil.brom@gmail.com> wrote:

> you may check the new regex implementation for python
> http://pypi.python.org/pypi/regex

Wow, I wasn't aware of such an effort.

At first reading, I'm amused by the concept of "strict fuzzy matching".

Those of us with long memories will be confused by the name of the 
module, however.  To me, "regex" is the old version, and "re" is the new 
(http://docs.python.org/release/1.5.1/lib/module-regex.html).  Now, it's 
going to be "regex" is either the really old, or the really new version, 
and "re" is what came in between.  I suppose the next iteration after 
this will be called "re" again :-)

I'm not sure I like the fuzzy matching stuff.  On the one hand, it can 
be useful.  On the other hand, people already complain about how 
difficult it can be to read regexes.  Once we add things like 
{i<=1,d<=1,s<=1,2i+2d+1s<=4} in the mix, it's going to be total line 
noise.

[toc] | [prev] | [next] | [standalone]

#35808

On 2012-12-30 17:32, Roy Smith wrote:
> In article <mailman.1467.1356885520.29569.python-list@python.org>,
>   Vlastimil Brom <vlastimil.brom@gmail.com> wrote:
>
>> you may check the new regex implementation for python
>> http://pypi.python.org/pypi/regex
>
> Wow, I wasn't aware of such an effort.
>
> At first reading, I'm amused by the concept of "strict fuzzy matching".
>
> Those of us with long memories will be confused by the name of the
> module, however.  To me, "regex" is the old version, and "re" is the new
> (http://docs.python.org/release/1.5.1/lib/module-regex.html).  Now, it's
> going to be "regex" is either the really old, or the really new version,
> and "re" is what came in between.  I suppose the next iteration after
> this will be called "re" again :-)
>
> I'm not sure I like the fuzzy matching stuff.  On the one hand, it can
> be useful.  On the other hand, people already complain about how
> difficult it can be to read regexes.  Once we add things like
> {i<=1,d<=1,s<=1,2i+2d+1s<=4} in the mix, it's going to be total line
> noise.
>
The request for fuzzy matching came after someone saw the TRE library:

http://laurikari.net/tre/

The difference is that the syntax is a bit clearer, IMHO. :-)

[toc] | [prev] | [next] | [standalone]

#35826

On 30Dec2012 12:32, Roy Smith <roy@panix.com> wrote:
| In article <mailman.1467.1356885520.29569.python-list@python.org>,
|  Vlastimil Brom <vlastimil.brom@gmail.com> wrote:
| > you may check the new regex implementation for python
| > http://pypi.python.org/pypi/regex
[...]
| I'm not sure I like the fuzzy matching stuff.  On the one hand, it can 
| be useful.  On the other hand, people already complain about how 
| difficult it can be to read regexes.  Once we add things like 
| {i<=1,d<=1,s<=1,2i+2d+1s<=4} in the mix, it's going to be total line 
| noise.

One might argue that people are not obliged to use the fuzzy matching
syntax (or any other part, really). It should be a matter of taste.
Of course, some people have bad taste:-)

However if you need fuzzy matching, it is very handy if there's a library to
hand that offers it.

Cheers,
-- 
Cameron Simpson <cs@zip.com.au>

Yes, sometimes Perl looks like line-noise to the uninitiated, but to the
seasoned Perl programmer, it looks like checksummed line-noise with a mission
in life.        - The Llama Book

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web