Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.forth > #59822

Re: RFD: Patching INCLUDED for UTF-8

Show all headers | View raw

On 5/25/2017 10:32, Anton Ertl wrote:
> Brad Eckert <hwfwguy@gmail.com> writes:
>> Now that the Unicode crowd has settled on UTF-8 for the most part (more than 50% of web pages now use UTF-8), it seems to be the new text file standard.
>>
>> My proposal is to have INCLUDED look at the first three bytes of the text file. If it starts with the 3-byte sequence 0xEF,0xBB,0xBF then skip those three bytes. This is the byte-order mark of UTF-8 files.
> 
> UTF-8 does not need a byte-order mark (it has only one byte order).
> This sequence is some Windowsism.  Maybe it will die out like
> ones-complement, or maybe it will persist.  Should we put code in
> Windows Forth systems to ignore this sequence?  Maybe.  Should be put
> code in non-Windows Forth systems to ignore this sequence.  What for?
> Should we standardize ignoring this sequence?  Probably not.
> 
> An easy way to deal with it is to produce a noop word with that name,
> and have a space after that word in all files that start with this
> word.
> 
> - anton
> 

The Unicode consortium advice is here. 
http://unicode.org/faq/utf_bom.html#bom10. Point 3 is worth noting.

3. Some byte oriented protocols expect ASCII characters at the beginning 
of a file. If UTF-8 is used with these protocols, use of the BOM as 
encoding form signature should be avoided.

-- 
Alex

Back to comp.lang.forth | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

RFD: Patching INCLUDED for UTF-8 Brad Eckert <hwfwguy@gmail.com> - 2017-05-24 16:17 -0700
  Re: RFD: Patching INCLUDED for UTF-8 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2017-05-25 09:32 +0000
    Re: RFD: Patching INCLUDED for UTF-8 Alex <alex@rivadpm.com> - 2017-05-25 12:09 +0100
      Re: RFD: Patching INCLUDED for UTF-8 Brad Eckert <hwfwguy@gmail.com> - 2017-05-25 09:40 -0700
    Re: RFD: Patching INCLUDED for UTF-8 Julian Fondren <julian.fondren@gmail.com> - 2017-05-25 09:51 -0700
      Re: RFD: Patching INCLUDED for UTF-8 Alex <alex@rivadpm.com> - 2017-05-25 21:28 +0100
        Re: RFD: Patching INCLUDED for UTF-8 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2017-05-26 09:39 +0000
          Re: RFD: Patching INCLUDED for UTF-8 Alex <alex@rivadpm.com> - 2017-05-26 13:27 +0100
            Re: RFD: Patching INCLUDED for UTF-8 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2017-05-26 14:56 +0000
            Re: RFD: Patching INCLUDED for UTF-8 Brad Eckert <hwfwguy@gmail.com> - 2017-05-31 21:55 -0700
              Re: RFD: Patching INCLUDED for UTF-8 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2017-06-01 06:23 +0000
      Re: RFD: Patching INCLUDED for UTF-8 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2017-05-26 09:45 +0000

csiph-web