Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.php > #9093 > unrolled thread

Rejecting Certain Non-ASCII Characters

Back to article view | Back to comp.lang.php

Contents

  Rejecting Certain Non-ASCII Characters Jim Higgins <ILikeMy@Privacy.bogus> - 2013-04-19 16:16 +0000
    Re: Rejecting Certain Non-ASCII Characters Lew Pitcher <lpitcher@teksavvy.com> - 2013-04-19 12:29 -0400
    Re: Rejecting Certain Non-ASCII Characters Daniel Pitts <newsgroup.nospam@virtualinfinity.net> - 2013-04-19 09:30 -0700
      Re: Rejecting Certain Non-ASCII Characters Jim Higgins <ILikeMy@Privacy.bogus> - 2013-04-19 21:38 +0000
        Re: Rejecting Certain Non-ASCII Characters Jim Higgins <ILikeMy@Privacy.bogus> - 2013-04-19 22:11 +0000
    Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-19 12:48 -0400
      Re: Rejecting Certain Non-ASCII Characters Jim Higgins <ILikeMy@Privacy.bogus> - 2013-04-19 21:46 +0000
        Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-19 18:00 -0400
        Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 01:23 +0200
    Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-19 19:38 +0200
      Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-19 22:33 +0200
        Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-19 23:08 +0200
          Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 01:12 +0200
            Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-22 13:25 +0200
              Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-22 13:29 +0200
          Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 14:08 +0200
            Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 17:52 +0200
      Re: Rejecting Certain Non-ASCII Characters Jim Higgins <ILikeMy@Privacy.bogus> - 2013-04-19 22:06 +0000
        Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-20 00:26 +0200
          Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-19 19:53 -0400
            Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-20 02:36 +0200
              Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-19 20:58 -0400
                Re: Rejecting Certain Non-ASCII Characters Christoph Becker <cmbecker69@gmx.de> - 2013-04-20 03:34 +0200
    Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-19 22:19 +0200
    Re: Rejecting Certain Non-ASCII Characters "M. Strobel" <sorry_no_mail_here@nowhere.dee> - 2013-04-20 10:04 +0200
      Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 14:15 +0200
        Re: Rejecting Certain Non-ASCII Characters "M. Strobel" <sorry_no_mail_here@nowhere.dee> - 2013-04-20 14:26 +0200
          Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 14:34 +0200
          Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-20 14:41 +0200
            Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-20 08:43 -0400
              Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-20 14:55 +0200
                Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-20 09:04 -0400
                  Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-20 21:29 +0200
                    Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-20 20:03 -0400
                      Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-21 06:51 +0200
                        Re: Rejecting Certain Non-ASCII Characters Arno Welzel <usenet@arnowelzel.de> - 2013-04-21 06:57 +0200
                          Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-21 08:45 -0400
                            Re: Rejecting Certain Non-ASCII Characters Tim Streater <timstreater@greenbee.net> - 2013-04-21 15:09 +0100
                              Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-21 10:29 -0400
                        Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-21 08:44 -0400
                Re: Rejecting Certain Non-ASCII Characters The Natural Philosopher <tnp@invalid.invalid> - 2013-04-20 15:07 +0100
                  Re: Rejecting Certain Non-ASCII Characters Jerry Stuckle <jstucklex@attglobal.net> - 2013-04-20 20:06 -0400
            Re: Rejecting Certain Non-ASCII Characters "M. Strobel" <sorry_no_mail_here@nowhere.dee> - 2013-04-20 17:43 +0200
              Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 18:01 +0200
                Re: Rejecting Certain Non-ASCII Characters "M. Strobel" <sorry_no_mail_here@nowhere.dee> - 2013-04-20 18:17 +0200
                  Address munging (was: Rejecting Certain Non-ASCII Characters) Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-20 18:23 +0200
              OT: Address munging (was: Re: Rejecting Certain Non-ASCII Characters) Arno Welzel <usenet@arnowelzel.de> - 2013-04-20 21:25 +0200
              Re: Rejecting Certain Non-ASCII Characters Denis McMahon <denismfmcmahon@gmail.com> - 2013-04-20 22:06 +0000
                Re: Rejecting Certain Non-ASCII Characters Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-22 14:10 +0200

Page 1 of 3  [1] 2 3  Next page →

#9093 — Rejecting Certain Non-ASCII Characters

I have a problem with people entering a slashed zero vs a standard
ASCII zero into HTML forms intended to store data in a MySQL database.

Is there a simple way in PHP to restrict input to the ASCII Character
set, specifically hex 0x20 - 0x7E ?  Or a simple way to detect
characters outside this range before committing them to the database?

My mind says it should be easy to detect and that I should be able to
do it myself, but for some reason I'm drawing a huge blank.

Thank you.

[toc] | [next] | [standalone]

#9094

On Friday 19 April 2013 12:16, in comp.lang.php, ILikeMy@Privacy.bogus
wrote:

> 
> I have a problem with people entering a slashed zero vs a standard
> ASCII zero into HTML forms intended to store data in a MySQL database.
> 
> Is there a simple way in PHP to restrict input to the ASCII Character
> set, specifically hex 0x20 - 0x7E ? 

http://www.w3.org/TR/html401/interact/forms.html#adef-accept-charset

[snip]

-- 
Lew Pitcher
"In Skills, We Trust"

[toc] | [prev] | [next] | [standalone]

#9095

On 4/19/13 9:16 AM, Jim Higgins wrote:
>
> I have a problem with people entering a slashed zero vs a standard
> ASCII zero into HTML forms intended to store data in a MySQL database.
>
> Is there a simple way in PHP to restrict input to the ASCII Character
> set, specifically hex 0x20 - 0x7E ?  Or a simple way to detect
> characters outside this range before committing them to the database?
>
> My mind says it should be easy to detect and that I should be able to
> do it myself, but for some reason I'm drawing a huge blank.
>
> Thank you.
>

Are they literally putting "\0", or do you mean it is sending the byte 0 
  (where "0" is the byte 48)?

If they are putting "\0", then this indicates you are not escaping the 
string appropriately. Don't filter it, escape it.

If it is the byte 0, then that is a very unusual thing for them to submit.

[toc] | [prev] | [next] | [standalone]

#9111

On Fri, 19 Apr 2013 09:30:54 -0700, in
<2ject.401871$Nq4.359760@newsfe21.iad>, Daniel Pitts
<newsgroup.nospam@virtualinfinity.net> wrote:

>On 4/19/13 9:16 AM, Jim Higgins wrote:
>>
>> I have a problem with people entering a slashed zero vs a standard
>> ASCII zero into HTML forms intended to store data in a MySQL database.
>>
>> Is there a simple way in PHP to restrict input to the ASCII Character
>> set, specifically hex 0x20 - 0x7E ?  Or a simple way to detect
>> characters outside this range before committing them to the database?
>>
>> My mind says it should be easy to detect and that I should be able to
>> do it myself, but for some reason I'm drawing a huge blank.
>>
>> Thank you.
>>
>
>Are they literally putting "\0", or do you mean it is sending the byte 0 
>  (where "0" is the byte 48)?
>
>If they are putting "\0", then this indicates you are not escaping the 
>string appropriately. Don't filter it, escape it.
>
>If it is the byte 0, then that is a very unusual thing for them to submit.


Where I want to see byte 48 (decimal) they are sending the code for a
zero with a slash thru it.  When displaying the data on an HTML page
it appears as "A~" consisting of 0x41 0x7E hex (65 254 decimal.)

[toc] | [prev] | [next] | [standalone]

#9117

On Fri, 19 Apr 2013 21:38:31 +0000, in
<etd3n8paphvs1mp97ts8povdkaecg9u5in@4ax.com>, Jim Higgins
<ILikeMy@Privacy.bogus> wrote:

>On Fri, 19 Apr 2013 09:30:54 -0700, in
><2ject.401871$Nq4.359760@newsfe21.iad>, Daniel Pitts
><newsgroup.nospam@virtualinfinity.net> wrote:
>
>>On 4/19/13 9:16 AM, Jim Higgins wrote:
>>>
>>> I have a problem with people entering a slashed zero vs a standard
>>> ASCII zero into HTML forms intended to store data in a MySQL database.
>>>
>>> Is there a simple way in PHP to restrict input to the ASCII Character
>>> set, specifically hex 0x20 - 0x7E ?  Or a simple way to detect
>>> characters outside this range before committing them to the database?
>>>
>>> My mind says it should be easy to detect and that I should be able to
>>> do it myself, but for some reason I'm drawing a huge blank.
>>>
>>> Thank you.
>>>
>>
>>Are they literally putting "\0", or do you mean it is sending the byte 0 
>>  (where "0" is the byte 48)?
>>
>>If they are putting "\0", then this indicates you are not escaping the 
>>string appropriately. Don't filter it, escape it.
>>
>>If it is the byte 0, then that is a very unusual thing for them to submit.
>
>
>Where I want to see byte 48 (decimal) they are sending the code for a
>zero with a slash thru it.  When displaying the data on an HTML page
>it appears as "A~" consisting of 0x41 0x7E hex (65 254 decimal.)


Make that 65 126 decimal.

[toc] | [prev] | [next] | [standalone]

#9098

On 4/19/2013 12:16 PM, Jim Higgins wrote:
>
> I have a problem with people entering a slashed zero vs a standard
> ASCII zero into HTML forms intended to store data in a MySQL database.
>
> Is there a simple way in PHP to restrict input to the ASCII Character
> set, specifically hex 0x20 - 0x7E ?  Or a simple way to detect
> characters outside this range before committing them to the database?
>
> My mind says it should be easy to detect and that I should be able to
> do it myself, but for some reason I'm drawing a huge blank.
>
> Thank you.
>

Short of parsing the string some way, no.

If it's a short string, you could loop through it character by 
character.  Longer strings might benefit from a regex.

But be aware just because YOU are working with an ASCII charset, not 
everyone is.

-- 
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

[toc] | [prev] | [next] | [standalone]

#9112

On Fri, 19 Apr 2013 12:48:14 -0400, in <kkrsb6$lec$1@dont-email.me>,
Jerry Stuckle <jstucklex@attglobal.net> wrote:

>On 4/19/2013 12:16 PM, Jim Higgins wrote:
>>
>> I have a problem with people entering a slashed zero vs a standard
>> ASCII zero into HTML forms intended to store data in a MySQL database.
>>
>> Is there a simple way in PHP to restrict input to the ASCII Character
>> set, specifically hex 0x20 - 0x7E ?  Or a simple way to detect
>> characters outside this range before committing them to the database?
>>
>> My mind says it should be easy to detect and that I should be able to
>> do it myself, but for some reason I'm drawing a huge blank.
>>
>> Thank you.
>>
>
>Short of parsing the string some way, no.
>
>If it's a short string, you could loop through it character by 
>character.  Longer strings might benefit from a regex.


The strings are very short, 1 - 7 characters.

>But be aware just because YOU are working with an ASCII charset, not 
>everyone is.

I'm not looking for worldwide compatibility.  My primary concern is
with the annoying affectation a few users have of using a slashed zero
in place of a just plain zero.  None are using a non-ASCII character
set otherwise.

[toc] | [prev] | [next] | [standalone]

#9115

On 4/19/2013 5:46 PM, Jim Higgins wrote:
> On Fri, 19 Apr 2013 12:48:14 -0400, in <kkrsb6$lec$1@dont-email.me>,
> Jerry Stuckle <jstucklex@attglobal.net> wrote:
>
>> On 4/19/2013 12:16 PM, Jim Higgins wrote:
>>>
>>> I have a problem with people entering a slashed zero vs a standard
>>> ASCII zero into HTML forms intended to store data in a MySQL database.
>>>
>>> Is there a simple way in PHP to restrict input to the ASCII Character
>>> set, specifically hex 0x20 - 0x7E ?  Or a simple way to detect
>>> characters outside this range before committing them to the database?
>>>
>>> My mind says it should be easy to detect and that I should be able to
>>> do it myself, but for some reason I'm drawing a huge blank.
>>>
>>> Thank you.
>>>
>>
>> Short of parsing the string some way, no.
>>
>> If it's a short string, you could loop through it character by
>> character.  Longer strings might benefit from a regex.
>
>
> The strings are very short, 1 - 7 characters.
>
>> But be aware just because YOU are working with an ASCII charset, not
>> everyone is.
>
> I'm not looking for worldwide compatibility.  My primary concern is
> with the annoying affectation a few users have of using a slashed zero
> in place of a just plain zero.  None are using a non-ASCII character
> set otherwise.
>

I understand - but YOU have to be aware that people ARE using different 
charsets, and handle it according to what YOUR needs are.

-- 
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

[toc] | [prev] | [next] | [standalone]

#9122

Jim Higgins wrote:

> Jerry Stuckle <jstucklex@attglobal.net> wrote:
>> But be aware just because YOU are working with an ASCII charset, not
>> everyone is.

ACK.

> I'm not looking for worldwide compatibility.

On the World Wide Web, most certainly you should.  US-ASCII is simply 
obsolete there, because it hinders more than it helps.  The default 
character encoding for HTTP/1.0 and later is ISO-8859-1 (which AFAIK is 
PHP's default, too), and using UTF-8 or UTF-16 becomes increasingly common 
both on the Web and in current operating systems (see e. g. the percent 
encoding in RFC 3986 how to handle that).  Most recently, IDNs have been 
introduced.

PHP is fully equipped to deal with that (through functions), even though it 
has no native Unicode string support.

> My primary concern is with the annoying affectation a few users have of
> using a slashed zero in place of a just plain zero.  None are using a non
> ASCII character set otherwise.

One problem with helping you with your problem is that the term “slashed 
zero” actually is _not_ well-defined.


PointedEars
-- 
var bugRiddenCrashPronePieceOfJunk = (
    navigator.userAgent.indexOf('MSIE 5') != -1
    && navigator.userAgent.indexOf('Mac') != -1
)  // Plone, register_function.js:16

[toc] | [prev] | [next] | [standalone]

#9099

Jim Higgins wrote:
> I have a problem with people entering a slashed zero vs a standard
> ASCII zero into HTML forms intended to store data in a MySQL database.

Is it really a slashed zero (U+0030 U+0338) they're entering, or do they
enter some similar looking character such as the Danish Ø?  In the
former case you can simply replace the slashed zero with a standard
zero.  Assuming UTF-8 encoding:

  $input = str_replace('\xCC\xB8', '', $input);

> Is there a simple way in PHP to restrict input to the ASCII Character
> set, specifically hex 0x20 - 0x7E ?  Or a simple way to detect
> characters outside this range before committing them to the database?

If you're dealing with a numeric column, you may consider checking for
is_numeric().

-- 
Christoph M. Becker

[toc] | [prev] | [next] | [standalone]

#9109

Christoph Becker wrote:

> Jim Higgins wrote:
>> I have a problem with people entering a slashed zero vs a standard
>> ASCII zero into HTML forms intended to store data in a MySQL database.
> 
> Is it really a slashed zero (U+0030 U+0338) they're entering,

Could also be U+0030 U+337 or any other (allowed composition) of the at 
least 65536 Unicode characters that look(s) similar.  For example, it could 
be U+2205 EMPTY SET.

> or do they enter some similar looking character such as the Danish Ø?

That character, U+00D8 LATIN CAPITAL LETTER O WITH STROKE *and* its 
lowercase counterpart, U+00F8 LATIN SMALL LETTER O WITH STROKE, are present 
at least in Danish, Norwegian, and Faroese; They are also used in the 
International Phonetic Alphabet (IPA).

> In the former case you can simply replace the slashed zero with a standard
> zero.  Assuming UTF-8 encoding:
> 
>   $input = str_replace('\xCC\xB8', '', $input);

You do not consider that, ambiguity aside, even in UTF-8 there are *several* 
ways to produce Unicode characters.  See: Unicode Normalization Forms.

>> Is there a simple way in PHP to restrict input to the ASCII Character
>> set, specifically hex 0x20 - 0x7E ?  Or a simple way to detect
>> characters outside this range before committing them to the database?
> 
> If you're dealing with a numeric column, you may consider checking for
> is_numeric().

There are also regular expressions.  Testing against '/[^\0x00-\x7F]/', and 
rejecting anything that matches, appears to be the best approach here.  If 
characters actually should be converted, iconv() should be used instead of a 
hardcoded conversion.

However, it is better in the long term to convert the MySQL database (or the 
relevant table and column) to utf8_general_ci, and upgrade MySQL if 
necessary (character sets and collations were not supported before MySQL 5; 
the current stable version is 5.6).

<http://php.net/pcre>
<http://dev.mysql.com/>


PointedEars
-- 
Prototype.js was written by people who don't know javascript for people
who don't know javascript. People who don't know javascript are not
the best source of advice on designing systems that use javascript.
  -- Richard Cornford, cljs, <f806at$ail$1$8300dec7@news.demon.co.uk>

[toc] | [prev] | [next] | [standalone]

#9110

Thomas 'PointedEars' Lahn wrote:
> Christoph Becker wrote:
> 
>> Jim Higgins wrote:
>>> I have a problem with people entering a slashed zero vs a standard
>>> ASCII zero into HTML forms intended to store data in a MySQL database.
>>
>> Is it really a slashed zero (U+0030 U+0338) they're entering,
> 
> Could also be U+0030 U+337 or any other (allowed composition) of the at 
> least 65536 Unicode characters that look(s) similar.  For example, it could 
> be U+2205 EMPTY SET.

Which I was referring to in a abbreviated form in the rest of the
sentence. :)

>> or do they enter some similar looking character such as the Danish Ø?
> 
> That character, U+00D8 LATIN CAPITAL LETTER O WITH STROKE *and* its 
> lowercase counterpart, U+00F8 LATIN SMALL LETTER O WITH STROKE, are present 
> at least in Danish, Norwegian, and Faroese; They are also used in the 
> International Phonetic Alphabet (IPA).
> 
>> In the former case you can simply replace the slashed zero with a standard
>> zero.  Assuming UTF-8 encoding:
>>
>>   $input = str_replace('\xCC\xB8', '', $input);
> 
> You do not consider that, ambiguity aside, even in UTF-8 there are *several* 
> ways to produce Unicode characters.  See: Unicode Normalization Forms.

If I'm not mistaken here, this str_replace() will work for NFD and NFC,
but indeed I had not considered NFKD and NFKC.  Thanks for pointing this
out.

>>> Is there a simple way in PHP to restrict input to the ASCII Character
>>> set, specifically hex 0x20 - 0x7E ?  Or a simple way to detect
>>> characters outside this range before committing them to the database?
>>
>> If you're dealing with a numeric column, you may consider checking for
>> is_numeric().
> 
> There are also regular expressions.  Testing against '/[^\0x00-\x7F]/', and 
> rejecting anything that matches, appears to be the best approach here.  

For a numeric column?  Why not at least trim down the possibilites of
setting illegal values for the SQL statement?

>                                                                         If 
> characters actually should be converted, iconv() should be used instead of a 
> hardcoded conversion.
> 
> However, it is better in the long term to convert the MySQL database (or the 
> relevant table and column) to utf8_general_ci, and upgrade MySQL if 
> necessary (character sets and collations were not supported before MySQL 5; 
> the current stable version is 5.6).
> 
> <http://php.net/pcre>
> <http://dev.mysql.com/>

-- 
Christoph M. Becker

[toc] | [prev] | [next] | [standalone]

#9121

Christoph Becker wrote:

> Thomas 'PointedEars' Lahn wrote:
>> Christoph Becker wrote:
>>> Jim Higgins wrote:
>>>> Is there a simple way in PHP to restrict input to the ASCII Character
>>>> set, specifically hex 0x20 - 0x7E ?  Or a simple way to detect
>>>> characters outside this range before committing them to the database?
>>>
>>> If you're dealing with a numeric column, you may consider checking for
>>> is_numeric().
>> 
>> There are also regular expressions.  Testing against '/[^\0x00-\x7F]/',
>> and rejecting anything that matches, appears to be the best approach
>> here.
> 
> For a numeric column?  Why not at least trim down the possibilites of
      ^^^^^^^^^^^^^^^^
> setting illegal values for the SQL statement?

A numeric column is *your* idea.  The questions I have answered were the 
ones that were *actually* asked:

| Is there a simple way in PHP to restrict input to the ASCII Character
| set, specifically hex 0x20 - 0x7E ?  Or a simple way to detect
| characters outside this range before committing them to the database?

The character at US-ASCII code point 0x7E is _not_ a digit in any number 
base.


PointedEars 
-- 
Danny Goodman's books are out of date and teach practices that are
positively harmful for cross-browser scripting.
  -- Richard Cornford, cljs, <cife6q$253$1$8300dec7@news.demon.co.uk> (2004)

[toc] | [prev] | [next] | [standalone]

#9175

Thomas 'PointedEars' Lahn wrote:
> A numeric column is *your* idea.  The questions I have answered were the 
> ones that were *actually* asked:
> 
> | Is there a simple way in PHP to restrict input to the ASCII Character
> | set, specifically hex 0x20 - 0x7E ?  Or a simple way to detect
> | characters outside this range before committing them to the database?

The OP had reported that users sometimes enter a slashed zero instead of
a "normal" zero.  In a column of type "char" or "varchar" this might not
matter too much, so I thought the problem might be related to numeric
columns instead. Therefore, I had written:

| If you're dealing with a numeric column, [...]
  ^^

-- 
Christoph M. Becker

[toc] | [prev] | [next] | [standalone]

#9176

Christoph Becker wrote:

> Thomas 'PointedEars' Lahn wrote:
>> A numeric column is *your* idea.  The questions I have answered were the
>> ones that were *actually* asked:
>> 
>> | Is there a simple way in PHP to restrict input to the ASCII Character
>> | set, specifically hex 0x20 - 0x7E ?  Or a simple way to detect
>> | characters outside this range before committing them to the database?
> 
> The OP had reported that users sometimes enter a slashed zero instead of
> a "normal" zero.  In a column of type "char" or "varchar" this might not
> matter too much, so I thought the problem might be related to numeric
> columns instead. Therefore, I had written:
> 
> | If you're dealing with a numeric column, [...]
>   ^^

And again, I was _not_ referring to that.  I wrote:

>>> Testing against '/[^\0x00-\x7F]/', and rejecting anything that matches,
>>> appears to be the best approach here.
                  ^^^^^^^^^^^^^^^^^^^^^^^


PointedEars 
-- 
When all you know is jQuery, every problem looks $(olvable).

[toc] | [prev] | [next] | [standalone]

#9141

Christoph Becker wrote:

> Thomas 'PointedEars' Lahn wrote:
>> Christoph Becker wrote:
>>> In the former case you can simply replace the slashed zero with a
>>> standard zero.  Assuming UTF-8 encoding:
>>>
>>>   $input = str_replace('\xCC\xB8', '', $input);
>> 
>> You do not consider that, ambiguity aside, even in UTF-8 there are
>> *several*
>> ways to produce Unicode characters.  See: Unicode Normalization Forms.
> 
> If I'm not mistaken here, this str_replace() will work for NFD and NFC,
> but indeed I had not considered NFKD and NFKC.  Thanks for pointing this
> out.

[I admit that Unicode Normalization Forms are tricky.  Even I did not know 
the full range of their complexities before your posting, so thank you for 
making me look into it more thoroughly.]

AIUI, your approach will work reliably *only* for Normalization Form D 
(NFD), which is the result of “Canonical *D*ecomposition” [1].  
Normalization Form C (NFC) is the result of “Canonical Decomposition, 
followed by Canonical *C*omposition“ (ibid.)  That is, if there is a 
character in Unicode that is a composition of the used glyphs, then there 
will be no code sequence for a combination character for you to remove.  

[NFKD and NFKC are entirely different animals: The NFKD of U+1E9B (“ẛ”) 
U+0323 is *U+0073* (“s”) U+0323 U+0307; its NFKC is *U+1E69* (“ṩ”).]

For example, if the original character was U+212B ANGSTROM SIGN (“Å”), then 
with NFC there will be only a code sequence for U+00C5 LATIN CAPITAL LETTER 
A WITH RING ABOVE (“Å”), and you will not find a code sequence for U+030A 
COMBINING RING ABOVE that you could remove in order to get the US-ASCII-
compliant U+0041 LATIN CAPITAL LETTER A (“A”).


However, apparently iconv(), which I suggested, cannot help here either:

$ php -r 'echo iconv("UTF-8", "US-ASCII", "-\xE2\x84\xAB-") . "\n";'
PHP Notice:  iconv(): Detected an illegal character in input string in 
Command line code on line 1
PHP Stack trace:
PHP   1. {main}() Command line code:0
PHP   2. iconv() Command line code:1
-

$ php -r 'echo iconv("UTF-8", "US-ASCII//TRANSLIT", "-\xE2\x84\xAB-") . 
"\n";'
-?-

$ php -r 'echo iconv("UTF-8", "US-ASCII//IGNORE", "-\xE2\x84\xAB-") . "\n";'
PHP Notice:  iconv(): Detected an illegal character in input string in 
Command line code on line 1
PHP Stack trace:
PHP   1. {main}() Command line code:0
PHP   2. iconv() Command line code:1
--

(Expected: -A-)

It might be possible using recode_string(), but I have not found out yet 
how.  My PHP was not compiled “--with-recode” and I can only get recode(1) 
to print “"A” for “Ä”, which is not helpful here.


PointedEars
___________
[1] <http://www.unicode.org/reports/tr15/tr15-37.html>
-- 
Anyone who slaps a 'this page is best viewed with Browser X' label on
a Web page appears to be yearning for the bad old days, before the Web,
when you had very little chance of reading a document written on another
computer, another word processor, or another network. -- Tim Berners-Lee

[toc] | [prev] | [next] | [standalone]

#9156

Thomas 'PointedEars' Lahn wrote:

> However, apparently iconv(), which I suggested, cannot help here either:
> 
> $ php -r 'echo iconv("UTF-8", "US-ASCII", "-\xE2\x84\xAB-") . "\n";'
> PHP Notice:  iconv(): Detected an illegal character in input string in
> Command line code on line 1
> PHP Stack trace:
> PHP   1. {main}() Command line code:0
> PHP   2. iconv() Command line code:1
> -
> 
> $ php -r 'echo iconv("UTF-8", "US-ASCII//TRANSLIT", "-\xE2\x84\xAB-") .
> "\n";'
> -?-
> 
> $ php -r 'echo iconv("UTF-8", "US-ASCII//IGNORE", "-\xE2\x84\xAB-") .
> "\n";'
> PHP Notice:  iconv(): Detected an illegal character in input string in
> Command line code on line 1
> PHP Stack trace:
> PHP   1. {main}() Command line code:0
> PHP   2. iconv() Command line code:1
> --
> 
> (Expected: -A-)
> 
> It might be possible using recode_string(), but I have not found out yet
> how.  My PHP was not compiled “--with-recode” and I can only get recode(1)
> to print “"A” for “Ä”, which is not helpful here.

On the other hand:

$ php -r 'echo iconv("UTF-8", "US-ASCII//TRANSLIT", \
"-Ȧ-Århus-\x30\xCC\xB8-") . "\n";'
-A-AArhus-0-

So while iconv() does not do Unicode normalization, it can be helpful here 
(you can replace “AArhus” unambiguously with the correct alternative 
spelling for the city, “Aarhus”, for example).


PointedEars
-- 
> If you get a bunch of authors […] that state the same "best practices"
> in any programming language, then you can bet who is wrong or right...
Not with javascript. Nonsense propagates like wildfire in this field.
  -- Richard Cornford, comp.lang.javascript, 2011-11-14

[toc] | [prev] | [next] | [standalone]

#9116

On Fri, 19 Apr 2013 19:38:02 +0200, in
<kkrvda$od5$1@speranza.aioe.org>, Christoph Becker <cmbecker69@gmx.de>
wrote:

>Jim Higgins wrote:
>> I have a problem with people entering a slashed zero vs a standard
>> ASCII zero into HTML forms intended to store data in a MySQL database.
>
>Is it really a slashed zero (U+0030 U+0338) they're entering, or do they
>enter some similar looking character such as the Danish Ø?  In the
>former case you can simply replace the slashed zero with a standard
>zero.  Assuming UTF-8 encoding:
>
>  $input = str_replace('\xCC\xB8', '', $input);


It's usually 0x41 0x7E, but sometimes 0xD8.  Rather than do
replacement I'd just like to detect and give them an error message
telling them to use their keyboard zero vs Alt Key plus numeric pad to
create special characters.


>> Is there a simple way in PHP to restrict input to the ASCII Character
>> set, specifically hex 0x20 - 0x7E ?  Or a simple way to detect
>> characters outside this range before committing them to the database?
>
>If you're dealing with a numeric column, you may consider checking for
>is_numeric().


Thank you.  That will solve half the problem - the case where data is
numeric only.  The other half of the issue is string fields containing
mixed alpha/numeric.

[toc] | [prev] | [next] | [standalone]

#9119

Jim Higgins wrote:
> On Fri, 19 Apr 2013 19:38:02 +0200, in
> <kkrvda$od5$1@speranza.aioe.org>, Christoph Becker <cmbecker69@gmx.de>
> wrote:
> 
>> Jim Higgins wrote:
>>> I have a problem with people entering a slashed zero vs a standard
>>> ASCII zero into HTML forms intended to store data in a MySQL database.
>>
>> Is it really a slashed zero (U+0030 U+0338) they're entering, or do they
>> enter some similar looking character such as the Danish Ø?  In the
>> former case you can simply replace the slashed zero with a standard
>> zero.  Assuming UTF-8 encoding:
>>
>>  $input = str_replace('\xCC\xB8', '', $input);
> 
> 
> It's usually 0x41 0x7E, but sometimes 0xD8.  

0xD8 is Ø in ISO-8859-1 for example; I do not know which character
encoding represents the same or a similar character as 0x41 0x7E.
Anyway, ISTM you're missing to enforce a particular character encoding
for your document (see <http://www.w3.org/TR/html4/charset.html> for
HTML 4.01 documents).

>                                             Rather than do
> replacement I'd just like to detect and give them an error message
> telling them to use their keyboard zero vs Alt Key plus numeric pad to
> create special characters.
> 
> 
>>> Is there a simple way in PHP to restrict input to the ASCII Character
>>> set, specifically hex 0x20 - 0x7E ?  Or a simple way to detect
>>> characters outside this range before committing them to the database?
>>
>> If you're dealing with a numeric column, you may consider checking for
>> is_numeric().
> 
> 
> Thank you.  That will solve half the problem - the case where data is
> numeric only.  The other half of the issue is string fields containing
> mixed alpha/numeric.

Then you should have a look at the answers of Arno
(<news:5171A6C4.7010706@arnowelzel.de>) and Thomas
(<news:62120436.b0HzOcHmYY@PointedEars.de>), who recommend using
ctype_print() resp. the regular expression '/[^\0x00-\x7F]/'.

-- 
Christoph M. Becker

[toc] | [prev] | [next] | [standalone]

#9125

On 4/19/2013 6:26 PM, Christoph Becker wrote:
> Jim Higgins wrote:
>> On Fri, 19 Apr 2013 19:38:02 +0200, in
>> <kkrvda$od5$1@speranza.aioe.org>, Christoph Becker <cmbecker69@gmx.de>
>> wrote:
>>
>>> Jim Higgins wrote:
>>>> I have a problem with people entering a slashed zero vs a standard
>>>> ASCII zero into HTML forms intended to store data in a MySQL database.
>>>
>>> Is it really a slashed zero (U+0030 U+0338) they're entering, or do they
>>> enter some similar looking character such as the Danish Ø?  In the
>>> former case you can simply replace the slashed zero with a standard
>>> zero.  Assuming UTF-8 encoding:
>>>
>>>   $input = str_replace('\xCC\xB8', '', $input);
>>
>>
>> It's usually 0x41 0x7E, but sometimes 0xD8.
>
> 0xD8 is Ø in ISO-8859-1 for example; I do not know which character
> encoding represents the same or a similar character as 0x41 0x7E.
> Anyway, ISTM you're missing to enforce a particular character encoding
> for your document (see <http://www.w3.org/TR/html4/charset.html> for
> HTML 4.01 documents).
>

This is a recommendation only.  The browser is free to ignore it.  There 
is no way to force a browser to do anything in HTML.

>>                                              Rather than do
>> replacement I'd just like to detect and give them an error message
>> telling them to use their keyboard zero vs Alt Key plus numeric pad to
>> create special characters.
>>
>>
>>>> Is there a simple way in PHP to restrict input to the ASCII Character
>>>> set, specifically hex 0x20 - 0x7E ?  Or a simple way to detect
>>>> characters outside this range before committing them to the database?
>>>
>>> If you're dealing with a numeric column, you may consider checking for
>>> is_numeric().
>>
>>
>> Thank you.  That will solve half the problem - the case where data is
>> numeric only.  The other half of the issue is string fields containing
>> mixed alpha/numeric.
>
> Then you should have a look at the answers of Arno
> (<news:5171A6C4.7010706@arnowelzel.de>) and Thomas
> (<news:62120436.b0HzOcHmYY@PointedEars.de>), who recommend using
> ctype_print() resp. the regular expression '/[^\0x00-\x7F]/'.
>


-- 
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | comp.lang.php

csiph-web