Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #36980

Re: Anyone know anything about string encodings?

Path csiph.com!eternal-september.org!feeder.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From Joshua Cranmer 🐧 <Pidgeot18@verizon.invalid>
Newsgroups comp.lang.java.programmer
Subject Re: Anyone know anything about string encodings?
Date Sat, 22 Jul 2017 12:20:22 -0500
Organization A noiseless patient Spider
Lines 28
Message-ID <ol019b$1q4$1@dont-email.me> (permalink)
References <e5731f8f-201f-43fb-99f4-46caac570c41@googlegroups.com> <okr94s$e1e$1@dont-email.me> <oku3r0$m19$2@gioia.aioe.org>
Mime-Version 1.0
Content-Type text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding 8bit
Injection-Date Sat, 22 Jul 2017 17:16:27 -0000 (UTC)
Injection-Info mx02.eternal-september.org; posting-host="68d7f7cfe0fe2462fb8caa2b3190ee38"; logging-data="1860"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1/GD4skPvPmop+Fw3f5sTtmMdM1BAVFitI="
User-Agent Mozilla/5.0 (Windows NT 6.3; WOW64; rv:55.0) Gecko/20100101 Thunderbird/55.0
In-Reply-To <oku3r0$m19$2@gioia.aioe.org>
Content-Language en-US
Cancel-Lock sha1:inEQzO/yqnxoLDpQ9qnGhqOgy6c=
Xref csiph.com comp.lang.java.programmer:36980

Show key headers only | View raw

On 7/21/2017 6:47 PM, Arne VajhĂžj wrote:
> I think my suggested algorithm handle your case 3 more
> according to standards.

For HTML, the standard more or less says "if there's no charset from the 
document itself or from the Content-Type, do whatever you want to."

In practice, web browsers do different things. I believe IE tries to do 
autodetection. I know Firefox falls back onto a locale-dependent default 
encoding (e.g., en-US defaults to Windows-1252, but ja-JP would use 
Shift-JIS). I don't know about Chrome or Safari.

Testing for UTF-8 is pretty much a safe first step because a) everyone 
already recommends using UTF-8 as the default anyways and b) it's very 
distinctive in its output format to the point that virtually all 8-bit 
non-UTF-8 text will fail the UTF-8 parsing. ICU, e.g., will say that 
text is UTF-8 with 50% confidence if it sees just 1 UTF-8 sequence and 
no invalid sequences moving to 100% confidence upon the third valid 
sequence.
> 
> Arne
> 
> 


-- 
Beware of bugs in the above code; I have only proved it correct, not 
tried it. -- Donald E. Knuth

Back to comp.lang.java.programmer | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Anyone know anything about string encodings? Eric Douglas <e.d.programmer@gmail.com> - 2017-07-20 13:22 -0700
  Re: Anyone know anything about string encodings? Joshua Cranmer 🐧 <Pidgeot18@verizon.invalid> - 2017-07-20 17:03 -0500
    Re: Anyone know anything about string encodings? Eric Douglas <e.d.programmer@gmail.com> - 2017-07-21 06:07 -0700
    Re: Anyone know anything about string encodings? Arne VajhĂžj <arne@vajhoej.dk> - 2017-07-21 19:47 -0400
      Re: Anyone know anything about string encodings? Joshua Cranmer 🐧 <Pidgeot18@verizon.invalid> - 2017-07-22 12:20 -0500
        Re: Anyone know anything about string encodings? Arne VajhĂžj <arne@vajhoej.dk> - 2017-07-22 16:40 -0400
          Re: Anyone know anything about string encodings? Joshua Cranmer 🐧 <Pidgeot18@verizon.invalid> - 2017-07-24 09:27 -0500
            Re: Anyone know anything about string encodings? Arne VajhĂžj <arne@vajhoej.dk> - 2017-07-24 10:54 -0400
              Re: Anyone know anything about string encodings? Joshua Cranmer 🐧 <Pidgeot18@verizon.invalid> - 2017-07-24 16:04 -0500
                Re: Anyone know anything about string encodings? lewbloch@gmail.com - 2017-07-24 17:40 -0700
                Re: Anyone know anything about string encodings? Arne VajhĂžj <arne@vajhoej.dk> - 2017-07-25 09:16 -0400
                Re: Anyone know anything about string encodings? Jukka Lahtinen <jtfjdehf@hotmail.com.invalid> - 2017-07-25 11:50 +0300
                Re: Anyone know anything about string encodings? Jukka Lahtinen <jtfjdehf@hotmail.com.invalid> - 2017-07-25 11:51 +0300
                Re: Anyone know anything about string encodings? Joshua Cranmer 🐧 <Pidgeot18@verizon.invalid> - 2017-07-25 09:39 -0500
                Re: Anyone know anything about string encodings? Arne VajhĂžj <arne@vajhoej.dk> - 2017-07-25 14:34 -0400
                Re: Anyone know anything about string encodings? "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> - 2017-07-25 13:07 +0200
                Re: Anyone know anything about string encodings? Arne VajhĂžj <arne@vajhoej.dk> - 2017-07-25 09:12 -0400
                Re: Anyone know anything about string encodings? Jukka Lahtinen <jtfjdehf@hotmail.com.invalid> - 2017-07-25 18:05 +0300
                Re: Anyone know anything about string encodings? Arne VajhĂžj <arne@vajhoej.dk> - 2017-07-25 14:31 -0400
                Re: Anyone know anything about string encodings? lewbloch@gmail.com - 2017-07-26 11:23 -0700
  Re: Anyone know anything about string encodings? Eric Sosman <esosman@comcast-dot-net.invalid> - 2017-07-20 18:09 -0400
    Re: Anyone know anything about string encodings? Eric Douglas <e.d.programmer@gmail.com> - 2017-07-21 06:09 -0700
      Re: Anyone know anything about string encodings? Arne VajhĂžj <arne@vajhoej.dk> - 2017-07-21 19:46 -0400
  Re: Anyone know anything about string encodings? Arne VajhĂžj <arne@vajhoej.dk> - 2017-07-20 20:18 -0400

csiph-web