Path: csiph.com!eternal-september.org!feeder.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail From: =?UTF-8?Q?Joshua_Cranmer_=f0=9f=90=a7?= Newsgroups: comp.lang.java.programmer Subject: Re: Anyone know anything about string encodings? Date: Sat, 22 Jul 2017 12:20:22 -0500 Organization: A noiseless patient Spider Lines: 28 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Injection-Date: Sat, 22 Jul 2017 17:16:27 -0000 (UTC) Injection-Info: mx02.eternal-september.org; posting-host="68d7f7cfe0fe2462fb8caa2b3190ee38"; logging-data="1860"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1/GD4skPvPmop+Fw3f5sTtmMdM1BAVFitI=" User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:55.0) Gecko/20100101 Thunderbird/55.0 In-Reply-To: Content-Language: en-US Cancel-Lock: sha1:inEQzO/yqnxoLDpQ9qnGhqOgy6c= Xref: csiph.com comp.lang.java.programmer:36980 On 7/21/2017 6:47 PM, Arne Vajhøj wrote: > I think my suggested algorithm handle your case 3 more > according to standards. For HTML, the standard more or less says "if there's no charset from the document itself or from the Content-Type, do whatever you want to." In practice, web browsers do different things. I believe IE tries to do autodetection. I know Firefox falls back onto a locale-dependent default encoding (e.g., en-US defaults to Windows-1252, but ja-JP would use Shift-JIS). I don't know about Chrome or Safari. Testing for UTF-8 is pretty much a safe first step because a) everyone already recommends using UTF-8 as the default anyways and b) it's very distinctive in its output format to the point that virtually all 8-bit non-UTF-8 text will fail the UTF-8 parsing. ICU, e.g., will say that text is UTF-8 with 50% confidence if it sees just 1 UTF-8 sequence and no invalid sequences moving to 100% confidence upon the third valid sequence. > > Arne > > -- Beware of bugs in the above code; I have only proved it correct, not tried it. -- Donald E. Knuth