Path: csiph.com!aioe.org!.POSTED!not-for-mail
From: =?UTF-8?Q?Arne_Vajh=c3=b8j?= <arne@vajhoej.dk>
Newsgroups: comp.lang.java.programmer
Subject: Re: Anyone know anything about string encodings?
Date: Sat, 22 Jul 2017 16:40:20 -0400
Organization: Aioe.org NNTP Server
Lines: 33
Message-ID: <ol0d7j$adc$1@gioia.aioe.org>
References: <e5731f8f-201f-43fb-99f4-46caac570c41@googlegroups.com> <okr94s$e1e$1@dont-email.me> <oku3r0$m19$2@gioia.aioe.org> <ol019b$1q4$1@dont-email.me>
NNTP-Posting-Host: PBuS014lhzVyHjLbXmo/ZQ.user.gioia.aioe.org
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Complaints-To: abuse@aioe.org
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1
Content-Language: en-US
X-Notice: Filtered by postfilter v. 0.8.2
Xref: csiph.com comp.lang.java.programmer:36981

On 7/22/2017 1:20 PM, Joshua Cranmer 🐧 wrote:
> On 7/21/2017 6:47 PM, Arne Vajhøj wrote:
>> I think my suggested algorithm handle your case 3 more
>> according to standards.
> 
> For HTML, the standard more or less says "if there's no charset from the 
> document itself or from the Content-Type, do whatever you want to."
> 
> In practice, web browsers do different things. I believe IE tries to do 
> autodetection. I know Firefox falls back onto a locale-dependent default 
> encoding (e.g., en-US defaults to Windows-1252, but ja-JP would use 
> Shift-JIS). I don't know about Chrome or Safari.
> 
> Testing for UTF-8 is pretty much a safe first step because a) everyone 
> already recommends using UTF-8 as the default anyways and b) it's very 
> distinctive in its output format to the point that virtually all 8-bit 
> non-UTF-8 text will fail the UTF-8 parsing. ICU, e.g., will say that 
> text is UTF-8 with 50% confidence if it sees just 1 UTF-8 sequence and 
> no invalid sequences moving to 100% confidence upon the third valid 
> sequence.

ISO-8859-1 used to be the default for HTML pages.

But checking the HTML 5 specs I can not find it. It does say
that the default encoding for eastern Europe is ISO-8859-2,
windows-1251 (which is close to ISO-8859-5) etc.. But I can not
find a place where it specify ISO-8859-1 (or windows-1252)
to be default for western Europe.

????

Arne