Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!rt.uk.eu.org!newsfeed.xs4all.nl!newsfeed1.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Peter Otten <__peter__@web.de>
Subject: Re: Diacretical incensitive search
Date: Fri, 17 May 2013 10:30:04 +0200
Organization: None
References: <20130517085704.3f6609e8@pcolivier.chezmoi.net>
Mime-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8Bit
User-Agent: KNode/4.7.3
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.1783.1368779403.3114.python-list@python.org>
Lines: 55
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:45473

Olive wrote:

> One feature that seems to be missing in the re module (or any tools that I
> know for searching text) is "diacretical incensitive search". I would like
> to have a match for something like this:
> 
> re.match("franc", "français")
> 
> in about the same whay we can have a case incensitive search:
> 
> re.match("(?i)fran", "Français").
> 
> Another related and more general problem (in the sense that it could
> easily be used to solve the first problem) would be to translate a string
> removing any diacritical mark:
> 
> nodiac("Français") -> "Francais"
> 
> The algorithm to write such a function is trivial but there are a lot of
> mark we can put on a letter. It would be necessary to have the list of
> "a"'s with something on it. i.e. "à,á,ã", etc. and this for every letter.
> Trying to make such a list by hand would inevitably lead to some symbols
> forgotten (and would be tedious).

[Python3.3]

>>> unicodedata.normalize("NFKD", "Français").encode("ascii", 
"ignore").decode()
'Francais'

import sys
from collections import defaultdict
from unicodedata import name, normalize

d = defaultdict(list)
for i in range(sys.maxunicode):
    c = chr(i)
    n = normalize("NFKD", c)[0]
    if ord(n) < 128 and n.isalpha(): # optional
        d[n].append(c)

for k, v in d.items():
    if len(v) > 1:
        print(k, "".join(v))

See also <http://effbot.org/zone/unicode-convert.htm>

PS: Be warned that experiments on the console may be misleading:

>>> unicodedata.normalize("NFKD", "ç")
'c'
>>> ascii(_)
"'c\\u0327'"