Re: Rejecting Certain Non-ASCII Characters

Message-ID	<62120436.b0HzOcHmYY@PointedEars.de> (permalink)
From	Thomas 'PointedEars' Lahn <PointedEars@web.de>
Organization	PointedEars Software (PES)
Date	2013-04-19 22:33 +0200
Subject	Re: Rejecting Certain Non-ASCII Characters
Newsgroups	comp.lang.php
References	<qfq2n85a3pb4eegtf7buji0o8ern0r838m@4ax.com> <kkrvda$od5$1@speranza.aioe.org>

Show all headers | View raw

Christoph Becker wrote:

> Jim Higgins wrote:
>> I have a problem with people entering a slashed zero vs a standard
>> ASCII zero into HTML forms intended to store data in a MySQL database.
> 
> Is it really a slashed zero (U+0030 U+0338) they're entering,

Could also be U+0030 U+337 or any other (allowed composition) of the at 
least 65536 Unicode characters that look(s) similar.  For example, it could 
be U+2205 EMPTY SET.

> or do they enter some similar looking character such as the Danish Ø?

That character, U+00D8 LATIN CAPITAL LETTER O WITH STROKE *and* its 
lowercase counterpart, U+00F8 LATIN SMALL LETTER O WITH STROKE, are present 
at least in Danish, Norwegian, and Faroese; They are also used in the 
International Phonetic Alphabet (IPA).

> In the former case you can simply replace the slashed zero with a standard
> zero.  Assuming UTF-8 encoding:
> 
>   $input = str_replace('\xCC\xB8', '', $input);

You do not consider that, ambiguity aside, even in UTF-8 there are *several* 
ways to produce Unicode characters.  See: Unicode Normalization Forms.

>> Is there a simple way in PHP to restrict input to the ASCII Character
>> set, specifically hex 0x20 - 0x7E ?  Or a simple way to detect
>> characters outside this range before committing them to the database?
> 
> If you're dealing with a numeric column, you may consider checking for
> is_numeric().

There are also regular expressions.  Testing against '/[^\0x00-\x7F]/', and 
rejecting anything that matches, appears to be the best approach here.  If 
characters actually should be converted, iconv() should be used instead of a 
hardcoded conversion.

However, it is better in the long term to convert the MySQL database (or the 
relevant table and column) to utf8_general_ci, and upgrade MySQL if 
necessary (character sets and collations were not supported before MySQL 5; 
the current stable version is 5.6).

<http://php.net/pcre>
<http://dev.mysql.com/>

PointedEars
-- 
Prototype.js was written by people who don't know javascript for people
who don't know javascript. People who don't know javascript are not
the best source of advice on designing systems that use javascript.
  -- Richard Cornford, cljs, <f806at$ail$1$8300dec7@news.demon.co.uk>

Thread

Rejecting Certain Non-ASCII Characters Jim Higgins <ILikeMy@Privacy.bogus> - 2013-04-19 16:16 +0000
  Re: Rejecting Certain Non-ASCII Characters Lew Pitcher <lpitcher@teksavvy.com> - 2013-04-19 12:29 -0400
  Re: Rejecting Certain Non-ASCII Characters Daniel Pitts <newsgroup.nospam@virtualinfinity.net> - 2013-04-19 09:30 -0700
    Re: Rejecting Certain Non-ASCII Characters Jim Higgins <ILikeMy@Privacy.bogus> - 2013-04-19 21:38 +0000
      Re: Rejecting Certain Non-ASCII Characters Jim Higgins <ILikeMy@Privacy.bogus> - 2013-04-19 22:11 +0000
  Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-19 12:48 -0400
    Re: Rejecting Certain Non-ASCII Characters Jim Higgins <ILikeMy@Privacy.bogus> - 2013-04-19 21:46 +0000
      Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-19 18:00 -0400
      Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 01:23 +0200
  Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-19 19:38 +0200
    Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-19 22:33 +0200
      Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-19 23:08 +0200
        Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 01:12 +0200
          Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-22 13:25 +0200
            Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-22 13:29 +0200
        Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 14:08 +0200
          Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 17:52 +0200
    Re: Rejecting Certain Non-ASCII Characters Jim Higgins <ILikeMy@Privacy.bogus> - 2013-04-19 22:06 +0000
      Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-20 00:26 +0200
        Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-19 19:53 -0400
          Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-20 02:36 +0200
            Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-19 20:58 -0400
              Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-20 03:34 +0200
  Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-19 22:19 +0200
  Re: Rejecting Certain Non-ASCII Characters "M. Strobel" <sorry_no_mail_here@nowhere.dee> - 2013-04-20 10:04 +0200
    Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 14:15 +0200
      Re: Rejecting Certain Non-ASCII Characters "M. Strobel" <sorry_no_mail_here@nowhere.dee> - 2013-04-20 14:26 +0200
        Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 14:34 +0200
        Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-20 14:41 +0200
          Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-20 08:43 -0400
            Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-20 14:55 +0200
              Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-20 09:04 -0400
                Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-20 21:29 +0200
                Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-20 20:03 -0400
                Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-21 06:51 +0200
                Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-21 06:57 +0200
                Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-21 08:45 -0400
                Re: Rejecting Certain Non-ASCII Characters Tim Streater <timstreater@greenbee.net> - 2013-04-21 15:09 +0100
                Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-21 10:29 -0400
                Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-21 08:44 -0400
              Re: Rejecting Certain Non-ASCII Characters The Natural Philosopher <tnp@invalid.invalid> - 2013-04-20 15:07 +0100
                Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-20 20:06 -0400
          Re: Rejecting Certain Non-ASCII Characters "M. Strobel" <sorry_no_mail_here@nowhere.dee> - 2013-04-20 17:43 +0200
            Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 18:01 +0200
              Re: Rejecting Certain Non-ASCII Characters "M. Strobel" <sorry_no_mail_here@nowhere.dee> - 2013-04-20 18:17 +0200
                Address munging (was: Rejecting Certain Non-ASCII Characters) Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 18:23 +0200
            OT: Address munging (was: Re: Rejecting Certain Non-ASCII Characters) Arno Welzel <usenet@arnowelzel.de> - 2013-04-20 21:25 +0200
            Re: Rejecting Certain Non-ASCII Characters Denis McMahon <denismfmcmahon@gmail.com> - 2013-04-20 22:06 +0000
              Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-22 14:10 +0200

csiph-web