Re: Rejecting Certain Non-ASCII Characters

From	Christoph Becker <cmbecker69@gmx.de>
Newsgroups	comp.lang.php
Subject	Re: Rejecting Certain Non-ASCII Characters
Date	2013-04-19 23:08 +0200
Organization	Aioe.org NNTP Server
Message-ID	<kksbol$348$1@speranza.aioe.org> (permalink)
References	<qfq2n85a3pb4eegtf7buji0o8ern0r838m@4ax.com> <kkrvda$od5$1@speranza.aioe.org> <62120436.b0HzOcHmYY@PointedEars.de>

Show all headers | View raw

Thomas 'PointedEars' Lahn wrote:
> Christoph Becker wrote:
> 
>> Jim Higgins wrote:
>>> I have a problem with people entering a slashed zero vs a standard
>>> ASCII zero into HTML forms intended to store data in a MySQL database.
>>
>> Is it really a slashed zero (U+0030 U+0338) they're entering,
> 
> Could also be U+0030 U+337 or any other (allowed composition) of the at 
> least 65536 Unicode characters that look(s) similar.  For example, it could 
> be U+2205 EMPTY SET.

Which I was referring to in a abbreviated form in the rest of the
sentence. :)

>> or do they enter some similar looking character such as the Danish Ø?
> 
> That character, U+00D8 LATIN CAPITAL LETTER O WITH STROKE *and* its 
> lowercase counterpart, U+00F8 LATIN SMALL LETTER O WITH STROKE, are present 
> at least in Danish, Norwegian, and Faroese; They are also used in the 
> International Phonetic Alphabet (IPA).
> 
>> In the former case you can simply replace the slashed zero with a standard
>> zero.  Assuming UTF-8 encoding:
>>
>>   $input = str_replace('\xCC\xB8', '', $input);
> 
> You do not consider that, ambiguity aside, even in UTF-8 there are *several* 
> ways to produce Unicode characters.  See: Unicode Normalization Forms.

If I'm not mistaken here, this str_replace() will work for NFD and NFC,
but indeed I had not considered NFKD and NFKC.  Thanks for pointing this
out.

>>> Is there a simple way in PHP to restrict input to the ASCII Character
>>> set, specifically hex 0x20 - 0x7E ?  Or a simple way to detect
>>> characters outside this range before committing them to the database?
>>
>> If you're dealing with a numeric column, you may consider checking for
>> is_numeric().
> 
> There are also regular expressions.  Testing against '/[^\0x00-\x7F]/', and 
> rejecting anything that matches, appears to be the best approach here.  

For a numeric column?  Why not at least trim down the possibilites of
setting illegal values for the SQL statement?

>                                                                         If 
> characters actually should be converted, iconv() should be used instead of a 
> hardcoded conversion.
> 
> However, it is better in the long term to convert the MySQL database (or the 
> relevant table and column) to utf8_general_ci, and upgrade MySQL if 
> necessary (character sets and collations were not supported before MySQL 5; 
> the current stable version is 5.6).
> 
> <http://php.net/pcre>
> <http://dev.mysql.com/>

-- 
Christoph M. Becker

Thread

Rejecting Certain Non-ASCII Characters Jim Higgins <ILikeMy@Privacy.bogus> - 2013-04-19 16:16 +0000
  Re: Rejecting Certain Non-ASCII Characters Lew Pitcher <lpitcher@teksavvy.com> - 2013-04-19 12:29 -0400
  Re: Rejecting Certain Non-ASCII Characters Daniel Pitts <newsgroup.nospam@virtualinfinity.net> - 2013-04-19 09:30 -0700
    Re: Rejecting Certain Non-ASCII Characters Jim Higgins <ILikeMy@Privacy.bogus> - 2013-04-19 21:38 +0000
      Re: Rejecting Certain Non-ASCII Characters Jim Higgins <ILikeMy@Privacy.bogus> - 2013-04-19 22:11 +0000
  Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-19 12:48 -0400
    Re: Rejecting Certain Non-ASCII Characters Jim Higgins <ILikeMy@Privacy.bogus> - 2013-04-19 21:46 +0000
      Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-19 18:00 -0400
      Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 01:23 +0200
  Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-19 19:38 +0200
    Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-19 22:33 +0200
      Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-19 23:08 +0200
        Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 01:12 +0200
          Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-22 13:25 +0200
            Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-22 13:29 +0200
        Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 14:08 +0200
          Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 17:52 +0200
    Re: Rejecting Certain Non-ASCII Characters Jim Higgins <ILikeMy@Privacy.bogus> - 2013-04-19 22:06 +0000
      Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-20 00:26 +0200
        Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-19 19:53 -0400
          Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-20 02:36 +0200
            Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-19 20:58 -0400
              Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-20 03:34 +0200
  Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-19 22:19 +0200
  Re: Rejecting Certain Non-ASCII Characters "M. Strobel" <sorry_no_mail_here@nowhere.dee> - 2013-04-20 10:04 +0200
    Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 14:15 +0200
      Re: Rejecting Certain Non-ASCII Characters "M. Strobel" <sorry_no_mail_here@nowhere.dee> - 2013-04-20 14:26 +0200
        Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 14:34 +0200
        Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-20 14:41 +0200
          Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-20 08:43 -0400
            Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-20 14:55 +0200
              Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-20 09:04 -0400
                Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-20 21:29 +0200
                Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-20 20:03 -0400
                Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-21 06:51 +0200
                Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-21 06:57 +0200
                Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-21 08:45 -0400
                Re: Rejecting Certain Non-ASCII Characters Tim Streater <timstreater@greenbee.net> - 2013-04-21 15:09 +0100
                Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-21 10:29 -0400
                Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-21 08:44 -0400
              Re: Rejecting Certain Non-ASCII Characters The Natural Philosopher <tnp@invalid.invalid> - 2013-04-20 15:07 +0100
                Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-20 20:06 -0400
          Re: Rejecting Certain Non-ASCII Characters "M. Strobel" <sorry_no_mail_here@nowhere.dee> - 2013-04-20 17:43 +0200
            Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 18:01 +0200
              Re: Rejecting Certain Non-ASCII Characters "M. Strobel" <sorry_no_mail_here@nowhere.dee> - 2013-04-20 18:17 +0200
                Address munging (was: Rejecting Certain Non-ASCII Characters) Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 18:23 +0200
            OT: Address munging (was: Re: Rejecting Certain Non-ASCII Characters) Arno Welzel <usenet@arnowelzel.de> - 2013-04-20 21:25 +0200
            Re: Rejecting Certain Non-ASCII Characters Denis McMahon <denismfmcmahon@gmail.com> - 2013-04-20 22:06 +0000
              Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-22 14:10 +0200

csiph-web