Re: trying to strip out non ascii.. or rather convert non ascii

Date	2013-10-30 13:10 -0400
From	Ned Batchelder <ned@nedbatchelder.com>
Subject	Re: trying to strip out non ascii.. or rather convert non ascii
References	(5 earlier) <mailman.1702.1382970129.18130.python-list@python.org> <526f46a2$0$6512$c3e8da3$5496439d@news.astraweb.com> <e018a4c6-e7a5-4356-8929-e26a3fdcb75d@googlegroups.com> <mailman.1806.1383137592.18130.python-list@python.org> <d4e620ab-e42c-4939-92b4-0c5c62c0bc8b@googlegroups.com>
Newsgroups	comp.lang.python
Message-ID	<mailman.1819.1383153035.18130.python-list@python.org> (permalink)

Show all headers | View raw

On 10/30/13 12:08 PM, wxjmfauth@gmail.com wrote:
> Le mercredi 30 octobre 2013 13:44:47 UTC+1, Ned Batchelder a écrit :
>> On 10/30/13 4:49 AM, wxjmfauth@gmail.com wrote:
>>
>>> Le mardi 29 octobre 2013 06:24:50 UTC+1, Steven D'Aprano a écrit :
>>>> On Mon, 28 Oct 2013 09:23:41 -0500, Tim Chase wrote:
>>>>> On 2013-10-28 07:01, wxjmfauth@gmail.com wrote:
>>>>>>> Simply ignoring diactrics won't get you very far.
>>>>>> Right. As an example, these four French words : cote, côte, coté, côté
>>>>>> .
>>>>> Distinct words with distinct meanings, sure.
>>>>> But when a naïve (naive? ☺) person or one without the easy ability to
>>>>> enter characters with diacritics searches for "cote", I want to return
>>>>> possible matches containing any of your 4 examples.  It's slightly
>>>>> fuzzier if they search for "coté", in which case they may mean "coté" or
>>>>> they might mean be unable to figure out how to add a hat and want to
>>>>> type "côté". Though I'd rather get more results, even if it has some
>>>>> that only match fuzzily.
>>>> The right solution to that is to treat it no differently from other fuzzy
>>>> searches. A good search engine should be tolerant of spelling errors and
>>>> alternative spellings for any letter, not just those with diacritics.
>>>> Ideally, a good search engine would successfully match all three of
>>>> "naïve", "naive" and "niave", and it shouldn't rely on special handling
>>>> of diacritics.
>>> ------
>>> This is a non sense. The purpose of a diacritical mark is to
>>> make a letter a different letter. If a tool is supposed to
>>> match an ô, there is absolutely no reason to match something
>>> else.
>>> jmf
>>
>>
>> jmf, Tim Chase described his use case, and it seems reasonable to me.
>>
>> I'm not sure why you would describe it as nonsense.
>>
>>
>>
>> --Ned.
> --------
>
> My comment had nothing to do with Python, it was a
> general comment. A diacritical mark just makes a letter
> a different letter; a "ï " and a "i" are "as
> diferent" as a "a" from a "z". A diacritical mark
> is more than a simple ornementation.

Yes, we understand that.  Tim outlined a need that had to do with users' 
informal typing.  In his case, he needs to deal with that sloppiness.  
You can't simply insist that users be more precise.

Unicode is a way to represent text, and text gets used in many different 
ways.  Each of us has to acknowledge that our text needs may be 
different than someone else's.  jmf, I'm guessing from your comments 
over the last few months that you are doing detailed linguistic work 
with corpora in many languages.  That work leads to one style of Unicode 
use.  In your domain, it is "nonsense" to ignore diacriticals.

Other people do different kinds of work with Unicode, and that leads to 
different needs.  In Tim's system, it is important to ignore 
diacriticals.  You might not have a use personally for Tim's system.  
That doesn't make it nonsense.

--Ned.
>  From a unicode perspective.
> Unicode.org "knows", these chars a very important, that's
> the reason why they exist in two forms, precomposed and
> composed forms.
>
>  From a software perspective.
> Luckily for the end users, all the serious software
> are considering all these chars in an equal way. They
> are all belonging to the BMP plane. An "Ą" is treated
> as an "ê", same memory consumption, same performance,
> ==> very smooth software.
>
> jmf
>

Thread

trying to strip out non ascii.. or rather convert non ascii bruce <badouglas@gmail.com> - 2013-10-26 16:11 -0400
Re: trying to strip out non ascii.. or rather convert non ascii Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-10-26 22:24 +0000
Re: trying to strip out non ascii.. or rather convert non ascii Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-10-26 20:51 -0400
Re: trying to strip out non ascii.. or rather convert non ascii Roy Smith <roy@panix.com> - 2013-10-26 21:11 -0400
Re: trying to strip out non ascii.. or rather convert non ascii Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-10-27 02:05 +0000
Re: trying to strip out non ascii.. or rather convert non ascii Chris Angelico <rosuav@gmail.com> - 2013-10-27 13:15 +1100
Re: trying to strip out non ascii.. or rather convert non ascii Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-10-27 09:21 +0000
Re: trying to strip out non ascii.. or rather convert non ascii Tim Chase <python.list@tim.thechases.com> - 2013-10-26 20:41 -0500
Re: trying to strip out non ascii.. or rather convert non ascii Roy Smith <roy@panix.com> - 2013-10-26 21:54 -0400
Re: trying to strip out non ascii.. or rather convert non ascii Tim Chase <python.list@tim.thechases.com> - 2013-10-26 21:17 -0500
Re: trying to strip out non ascii.. or rather convert non ascii Nobody <nobody@nowhere.com> - 2013-10-27 03:21 +0000
Re: trying to strip out non ascii.. or rather convert non ascii wxjmfauth@gmail.com - 2013-10-28 07:01 -0700
Re: trying to strip out non ascii.. or rather convert non ascii Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-10-28 14:13 +0000
Re: trying to strip out non ascii.. or rather convert non ascii Tim Chase <python.list@tim.thechases.com> - 2013-10-28 09:23 -0500
Re: trying to strip out non ascii.. or rather convert non ascii Steven D'Aprano <steve@pearwood.info> - 2013-10-29 05:24 +0000
Re: trying to strip out non ascii.. or rather convert non ascii wxjmfauth@gmail.com - 2013-10-30 01:49 -0700
Re: trying to strip out non ascii.. or rather convert non ascii Ned Batchelder <ned@nedbatchelder.com> - 2013-10-30 08:44 -0400
Re: trying to strip out non ascii.. or rather convert non ascii wxjmfauth@gmail.com - 2013-10-30 09:08 -0700
Re: trying to strip out non ascii.. or rather convert non ascii Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-10-30 16:24 +0000
Re: trying to strip out non ascii.. or rather convert non ascii Ned Batchelder <ned@nedbatchelder.com> - 2013-10-30 13:10 -0400
Re: trying to strip out non ascii.. or rather convert non ascii Michael Torrie <torriem@gmail.com> - 2013-10-30 11:54 -0600
Re: trying to strip out non ascii.. or rather convert non ascii wxjmfauth@gmail.com - 2013-10-30 11:38 -0700
Re: trying to strip out non ascii.. or rather convert non ascii Roy Smith <roy@panix.com> - 2013-10-30 19:28 -0400
Re: trying to strip out non ascii.. or rather convert non ascii Tim Chase <python.list@tim.thechases.com> - 2013-10-31 06:46 -0500
Re: trying to strip out non ascii.. or rather convert non ascii Terry Reedy <tjreedy@udel.edu> - 2013-10-30 17:56 -0400
Re: trying to strip out non ascii.. or rather convert non ascii Steven D'Aprano <steve@pearwood.info> - 2013-10-31 07:10 +0000
Re: trying to strip out non ascii.. or rather convert non ascii Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-10-31 07:23 +0000
Re: trying to strip out non ascii.. or rather convert non ascii wxjmfauth@gmail.com - 2013-10-31 03:33 -0700
Re: trying to strip out non ascii.. or rather convert non ascii Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-11-01 07:16 +0000
Re: trying to strip out non ascii.. or rather convert non ascii wxjmfauth@gmail.com - 2013-11-01 02:00 -0700
Re: trying to strip out non ascii.. or rather convert non ascii Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-11-01 09:18 +0000
Re: trying to strip out non ascii.. or rather convert non ascii Steven D'Aprano <steve@pearwood.info> - 2013-10-29 05:22 +0000
Re: trying to strip out non ascii.. or rather convert non ascii wxjmfauth@gmail.com - 2013-10-29 08:38 -0700
Re: trying to strip out non ascii.. or rather convert non ascii Tim Chase <python.list@tim.thechases.com> - 2013-10-29 10:52 -0500
Re: trying to strip out non ascii.. or rather convert non ascii wxjmfauth@gmail.com - 2013-10-29 12:16 -0700
Re: trying to strip out non ascii.. or rather convert non ascii Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-10-29 19:54 +0000
Re: trying to strip out non ascii.. or rather convert non ascii Piet van Oostrum <piet@vanoostrum.org> - 2013-10-29 21:33 -0400
Re: trying to strip out non ascii.. or rather convert non ascii Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-10-30 09:19 +0000
Re: trying to strip out non ascii.. or rather convert non ascii Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-10-29 15:56 +0000
Re: trying to strip out non ascii.. or rather convert non ascii Chris Angelico <rosuav@gmail.com> - 2013-10-30 13:17 +1100
Re: trying to strip out non ascii.. or rather convert non ascii wxjmfauth@gmail.com - 2013-10-30 01:13 -0700
Re: trying to strip out non ascii.. or rather convert non ascii Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-10-30 15:25 +0000

csiph-web