Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.forth > #59869

Re: RFD: Patching INCLUDED for UTF-8

Show all headers | View raw

Alex <alex@rivadpm.com> writes:
>On 5/26/2017 10:39, Anton Ertl wrote:
>> Alex <alex@rivadpm.com> writes:
>>> Stupidly, Microsoft say this;
>>> https://msdn.microsoft.com/en-us/library/windows/desktop/dd374101(v=vs.85).aspx
>> 
>> |Because Unicode plain text is a sequence of 16-bit code values, it is
>> |sensitive to the byte ordering used when the text is written.
>> 
>> So Microsoft means UTF-16 when they write "Unicode".  For UTF-16, byte
>> order is an issue, so a byte-order mark is not completely superfluous.
>> Also, because UTF-16 is not ASCII-compatible on byte-addressed
>> machines, the main disadvantage of prepending a BOM does not play a
>> role (tools that work on 8-bit characters don't work on UTF-16 anyway,
>> BOM or no).  But these reasons don't transfer to UTF-8.
>> 
>> - anton
>> 
>
>The table has the UTF-8 "BOM" in it. It takes a fairly narrow reading of 
>the text to assume that they aren't referring to UTF-8 but only 
>UTF-longer as Unicode, and I suspect they aren't.

That may be the case, but the sentence I cited above, which is one of
three in the introductory paragraph, as well as the discussion of byte
order differences further on shows where there focus is.  UTF-8 is, at
best an afterthought on that page.  Given that they don't mention it
except in the table, my guess is that the table entry was added later,
after someone complained about the missing UTF-8 entry in the table.

>"many pieces of software on Microsoft Windows such as Notepad treat the 
>BOM as a required magic number rather than use heuristics. These tools 
>add a BOM when saving text as UTF-8, and cannot interpret UTF-8 unless 
>the BOM is present or the file contains only ASCII. Google Docs also 
>adds a BOM when converting a document to a plain text file for download."

That's probably a stronger reason for BOM problems than the web page
above.

- anton
-- 
M. Anton Ertl  http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
     New standard: http://www.forth200x.org/forth200x.html
   EuroForth 2017: http://www.euroforth.org/ef17/

Back to comp.lang.forth | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

RFD: Patching INCLUDED for UTF-8 Brad Eckert <hwfwguy@gmail.com> - 2017-05-24 16:17 -0700
  Re: RFD: Patching INCLUDED for UTF-8 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2017-05-25 09:32 +0000
    Re: RFD: Patching INCLUDED for UTF-8 Alex <alex@rivadpm.com> - 2017-05-25 12:09 +0100
      Re: RFD: Patching INCLUDED for UTF-8 Brad Eckert <hwfwguy@gmail.com> - 2017-05-25 09:40 -0700
    Re: RFD: Patching INCLUDED for UTF-8 Julian Fondren <julian.fondren@gmail.com> - 2017-05-25 09:51 -0700
      Re: RFD: Patching INCLUDED for UTF-8 Alex <alex@rivadpm.com> - 2017-05-25 21:28 +0100
        Re: RFD: Patching INCLUDED for UTF-8 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2017-05-26 09:39 +0000
          Re: RFD: Patching INCLUDED for UTF-8 Alex <alex@rivadpm.com> - 2017-05-26 13:27 +0100
            Re: RFD: Patching INCLUDED for UTF-8 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2017-05-26 14:56 +0000
            Re: RFD: Patching INCLUDED for UTF-8 Brad Eckert <hwfwguy@gmail.com> - 2017-05-31 21:55 -0700
              Re: RFD: Patching INCLUDED for UTF-8 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2017-06-01 06:23 +0000
      Re: RFD: Patching INCLUDED for UTF-8 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2017-05-26 09:45 +0000

csiph-web