Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #45473

Re: Diacretical incensitive search

Show key headers only | View raw

Olive wrote:

> One feature that seems to be missing in the re module (or any tools that I
> know for searching text) is "diacretical incensitive search". I would like
> to have a match for something like this:
> 
> re.match("franc", "français")
> 
> in about the same whay we can have a case incensitive search:
> 
> re.match("(?i)fran", "Français").
> 
> Another related and more general problem (in the sense that it could
> easily be used to solve the first problem) would be to translate a string
> removing any diacritical mark:
> 
> nodiac("Français") -> "Francais"
> 
> The algorithm to write such a function is trivial but there are a lot of
> mark we can put on a letter. It would be necessary to have the list of
> "a"'s with something on it. i.e. "à,á,ã", etc. and this for every letter.
> Trying to make such a list by hand would inevitably lead to some symbols
> forgotten (and would be tedious).

[Python3.3]

>>> unicodedata.normalize("NFKD", "Français").encode("ascii", 
"ignore").decode()
'Francais'

import sys
from collections import defaultdict
from unicodedata import name, normalize

d = defaultdict(list)
for i in range(sys.maxunicode):
    c = chr(i)
    n = normalize("NFKD", c)[0]
    if ord(n) < 128 and n.isalpha(): # optional
        d[n].append(c)

for k, v in d.items():
    if len(v) > 1:
        print(k, "".join(v))

See also <http://effbot.org/zone/unicode-convert.htm>

PS: Be warned that experiments on the console may be misleading:

>>> unicodedata.normalize("NFKD", "ç")
'c'
>>> ascii(_)
"'c\\u0327'"

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Diacretical incensitive search Olive <diolu.remove_this_part@bigfoot.com> - 2013-05-17 08:57 +0200
  Re: Diacretical incensitive search Petite Abeille <petite.abeille@gmail.com> - 2013-05-17 09:15 +0200
  Re: Diacretical incensitive search Peter Otten <__peter__@web.de> - 2013-05-17 10:30 +0200
    Re: Diacretical incensitive search Olive <diolu.remove_this_part@bigfoot.com> - 2013-05-17 17:37 +0200
      Re: Diacretical incensitive search jmfauth <wxjmfauth@gmail.com> - 2013-05-17 10:31 -0700
  Re: Diacretical incensitive search Jorgen Grahn <grahn+nntp@snipabacken.se> - 2013-05-20 09:10 +0000

csiph-web