Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.php > #9141

Re: Rejecting Certain Non-ASCII Characters

Message-ID <1392900.4dMhgHte9z@PointedEars.de> (permalink)
From Thomas 'PointedEars' Lahn <PointedEars@web.de>
Organization PointedEars Software (PES)
Date 2013-04-20 14:08 +0200
Subject Re: Rejecting Certain Non-ASCII Characters
Newsgroups comp.lang.php
References <qfq2n85a3pb4eegtf7buji0o8ern0r838m@4ax.com> <kkrvda$od5$1@speranza.aioe.org> <62120436.b0HzOcHmYY@PointedEars.de> <kksbol$348$1@speranza.aioe.org>

Show all headers | View raw

Christoph Becker wrote:

> Thomas 'PointedEars' Lahn wrote:
>> Christoph Becker wrote:
>>> In the former case you can simply replace the slashed zero with a
>>> standard zero.  Assuming UTF-8 encoding:
>>>
>>>   $input = str_replace('\xCC\xB8', '', $input);
>> 
>> You do not consider that, ambiguity aside, even in UTF-8 there are
>> *several*
>> ways to produce Unicode characters.  See: Unicode Normalization Forms.
> 
> If I'm not mistaken here, this str_replace() will work for NFD and NFC,
> but indeed I had not considered NFKD and NFKC.  Thanks for pointing this
> out.

[I admit that Unicode Normalization Forms are tricky.  Even I did not know 
the full range of their complexities before your posting, so thank you for 
making me look into it more thoroughly.]

AIUI, your approach will work reliably *only* for Normalization Form D 
(NFD), which is the result of “Canonical *D*ecomposition” [1].  
Normalization Form C (NFC) is the result of “Canonical Decomposition, 
followed by Canonical *C*omposition“ (ibid.)  That is, if there is a 
character in Unicode that is a composition of the used glyphs, then there 
will be no code sequence for a combination character for you to remove.  

[NFKD and NFKC are entirely different animals: The NFKD of U+1E9B (“ẛ”) 
U+0323 is *U+0073* (“s”) U+0323 U+0307; its NFKC is *U+1E69* (“ṩ”).]

For example, if the original character was U+212B ANGSTROM SIGN (“Å”), then 
with NFC there will be only a code sequence for U+00C5 LATIN CAPITAL LETTER 
A WITH RING ABOVE (“Å”), and you will not find a code sequence for U+030A 
COMBINING RING ABOVE that you could remove in order to get the US-ASCII-
compliant U+0041 LATIN CAPITAL LETTER A (“A”).


However, apparently iconv(), which I suggested, cannot help here either:

$ php -r 'echo iconv("UTF-8", "US-ASCII", "-\xE2\x84\xAB-") . "\n";'
PHP Notice:  iconv(): Detected an illegal character in input string in 
Command line code on line 1
PHP Stack trace:
PHP   1. {main}() Command line code:0
PHP   2. iconv() Command line code:1
-

$ php -r 'echo iconv("UTF-8", "US-ASCII//TRANSLIT", "-\xE2\x84\xAB-") . 
"\n";'
-?-

$ php -r 'echo iconv("UTF-8", "US-ASCII//IGNORE", "-\xE2\x84\xAB-") . "\n";'
PHP Notice:  iconv(): Detected an illegal character in input string in 
Command line code on line 1
PHP Stack trace:
PHP   1. {main}() Command line code:0
PHP   2. iconv() Command line code:1
--

(Expected: -A-)

It might be possible using recode_string(), but I have not found out yet 
how.  My PHP was not compiled “--with-recode” and I can only get recode(1) 
to print “"A” for “Ä”, which is not helpful here.


PointedEars
___________
[1] <http://www.unicode.org/reports/tr15/tr15-37.html>
-- 
Anyone who slaps a 'this page is best viewed with Browser X' label on
a Web page appears to be yearning for the bad old days, before the Web,
when you had very little chance of reading a document written on another
computer, another word processor, or another network. -- Tim Berners-Lee

Back to comp.lang.php | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Rejecting Certain Non-ASCII Characters Jim Higgins <ILikeMy@Privacy.bogus> - 2013-04-19 16:16 +0000
  Re: Rejecting Certain Non-ASCII Characters Lew Pitcher <lpitcher@teksavvy.com> - 2013-04-19 12:29 -0400
  Re: Rejecting Certain Non-ASCII Characters Daniel Pitts <newsgroup.nospam@virtualinfinity.net> - 2013-04-19 09:30 -0700
    Re: Rejecting Certain Non-ASCII Characters Jim Higgins <ILikeMy@Privacy.bogus> - 2013-04-19 21:38 +0000
      Re: Rejecting Certain Non-ASCII Characters Jim Higgins <ILikeMy@Privacy.bogus> - 2013-04-19 22:11 +0000
  Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-19 12:48 -0400
    Re: Rejecting Certain Non-ASCII Characters Jim Higgins <ILikeMy@Privacy.bogus> - 2013-04-19 21:46 +0000
      Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-19 18:00 -0400
      Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 01:23 +0200
  Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-19 19:38 +0200
    Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-19 22:33 +0200
      Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-19 23:08 +0200
        Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 01:12 +0200
          Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-22 13:25 +0200
            Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-22 13:29 +0200
        Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 14:08 +0200
          Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 17:52 +0200
    Re: Rejecting Certain Non-ASCII Characters Jim Higgins <ILikeMy@Privacy.bogus> - 2013-04-19 22:06 +0000
      Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-20 00:26 +0200
        Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-19 19:53 -0400
          Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-20 02:36 +0200
            Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-19 20:58 -0400
              Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-20 03:34 +0200
  Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-19 22:19 +0200
  Re: Rejecting Certain Non-ASCII Characters "M. Strobel" <sorry_no_mail_here@nowhere.dee> - 2013-04-20 10:04 +0200
    Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 14:15 +0200
      Re: Rejecting Certain Non-ASCII Characters "M. Strobel" <sorry_no_mail_here@nowhere.dee> - 2013-04-20 14:26 +0200
        Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 14:34 +0200
        Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-20 14:41 +0200
          Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-20 08:43 -0400
            Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-20 14:55 +0200
              Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-20 09:04 -0400
                Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-20 21:29 +0200
                Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-20 20:03 -0400
                Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-21 06:51 +0200
                Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-21 06:57 +0200
                Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-21 08:45 -0400
                Re: Rejecting Certain Non-ASCII Characters Tim Streater <timstreater@greenbee.net> - 2013-04-21 15:09 +0100
                Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-21 10:29 -0400
                Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-21 08:44 -0400
              Re: Rejecting Certain Non-ASCII Characters The Natural Philosopher <tnp@invalid.invalid> - 2013-04-20 15:07 +0100
                Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-20 20:06 -0400
          Re: Rejecting Certain Non-ASCII Characters "M. Strobel" <sorry_no_mail_here@nowhere.dee> - 2013-04-20 17:43 +0200
            Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 18:01 +0200
              Re: Rejecting Certain Non-ASCII Characters "M. Strobel" <sorry_no_mail_here@nowhere.dee> - 2013-04-20 18:17 +0200
                Address munging (was: Rejecting Certain Non-ASCII Characters) Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 18:23 +0200
            OT: Address munging (was: Re: Rejecting Certain Non-ASCII Characters) Arno Welzel <usenet@arnowelzel.de> - 2013-04-20 21:25 +0200
            Re: Rejecting Certain Non-ASCII Characters Denis McMahon <denismfmcmahon@gmail.com> - 2013-04-20 22:06 +0000
              Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-22 14:10 +0200

csiph-web