Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.forth > #59855

Re: RFD: Patching INCLUDED for UTF-8

Show all headers | View raw

On 5/26/2017 10:39, Anton Ertl wrote:
> Alex <alex@rivadpm.com> writes:
>> Stupidly, Microsoft say this;
>> https://msdn.microsoft.com/en-us/library/windows/desktop/dd374101(v=vs.85).aspx
> 
> |Because Unicode plain text is a sequence of 16-bit code values, it is
> |sensitive to the byte ordering used when the text is written.
> 
> So Microsoft means UTF-16 when they write "Unicode".  For UTF-16, byte
> order is an issue, so a byte-order mark is not completely superfluous.
> Also, because UTF-16 is not ASCII-compatible on byte-addressed
> machines, the main disadvantage of prepending a BOM does not play a
> role (tools that work on 8-bit characters don't work on UTF-16 anyway,
> BOM or no).  But these reasons don't transfer to UTF-8.
> 
> - anton
> 

The table has the UTF-8 "BOM" in it. It takes a fairly narrow reading of 
the text to assume that they aren't referring to UTF-8 but only 
UTF-longer as Unicode, and I suspect they aren't.

"Therefore, Unicode has defined a character (U+FEFF) and a noncharacter 
(U+FFFE) as byte order marks. They are mirror byte images of each other."

That's so wrong it's not even wrong. It's the byte encoding of U+FEFF 
that looks like FE FF or FF FE under UTF-16xx. U+FFFE is undefined, and 
is not some kind of BOM.

It's basically a train wreck. Wikipedia notes:

"many pieces of software on Microsoft Windows such as Notepad treat the 
BOM as a required magic number rather than use heuristics. These tools 
add a BOM when saving text as UTF-8, and cannot interpret UTF-8 unless 
the BOM is present or the file contains only ASCII. Google Docs also 
adds a BOM when converting a document to a plain text file for download."

-- 
Alex

Back to comp.lang.forth | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

RFD: Patching INCLUDED for UTF-8 Brad Eckert <hwfwguy@gmail.com> - 2017-05-24 16:17 -0700
  Re: RFD: Patching INCLUDED for UTF-8 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2017-05-25 09:32 +0000
    Re: RFD: Patching INCLUDED for UTF-8 Alex <alex@rivadpm.com> - 2017-05-25 12:09 +0100
      Re: RFD: Patching INCLUDED for UTF-8 Brad Eckert <hwfwguy@gmail.com> - 2017-05-25 09:40 -0700
    Re: RFD: Patching INCLUDED for UTF-8 Julian Fondren <julian.fondren@gmail.com> - 2017-05-25 09:51 -0700
      Re: RFD: Patching INCLUDED for UTF-8 Alex <alex@rivadpm.com> - 2017-05-25 21:28 +0100
        Re: RFD: Patching INCLUDED for UTF-8 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2017-05-26 09:39 +0000
          Re: RFD: Patching INCLUDED for UTF-8 Alex <alex@rivadpm.com> - 2017-05-26 13:27 +0100
            Re: RFD: Patching INCLUDED for UTF-8 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2017-05-26 14:56 +0000
            Re: RFD: Patching INCLUDED for UTF-8 Brad Eckert <hwfwguy@gmail.com> - 2017-05-31 21:55 -0700
              Re: RFD: Patching INCLUDED for UTF-8 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2017-06-01 06:23 +0000
      Re: RFD: Patching INCLUDED for UTF-8 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2017-05-26 09:45 +0000

csiph-web