Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #36970 > unrolled thread

Anyone know anything about string encodings?

Back to article view | Back to comp.lang.java.programmer

Contents

  Anyone know anything about string encodings? Eric Douglas <e.d.programmer@gmail.com> - 2017-07-20 13:22 -0700
    Re: Anyone know anything about string encodings? Joshua Cranmer 🐧 <Pidgeot18@verizon.invalid> - 2017-07-20 17:03 -0500
      Re: Anyone know anything about string encodings? Eric Douglas <e.d.programmer@gmail.com> - 2017-07-21 06:07 -0700
      Re: Anyone know anything about string encodings? Arne Vajhøj <arne@vajhoej.dk> - 2017-07-21 19:47 -0400
        Re: Anyone know anything about string encodings? Joshua Cranmer 🐧 <Pidgeot18@verizon.invalid> - 2017-07-22 12:20 -0500
          Re: Anyone know anything about string encodings? Arne Vajhøj <arne@vajhoej.dk> - 2017-07-22 16:40 -0400
            Re: Anyone know anything about string encodings? Joshua Cranmer 🐧 <Pidgeot18@verizon.invalid> - 2017-07-24 09:27 -0500
              Re: Anyone know anything about string encodings? Arne Vajhøj <arne@vajhoej.dk> - 2017-07-24 10:54 -0400
                Re: Anyone know anything about string encodings? Joshua Cranmer 🐧 <Pidgeot18@verizon.invalid> - 2017-07-24 16:04 -0500
                  Re: Anyone know anything about string encodings? lewbloch@gmail.com - 2017-07-24 17:40 -0700
                    Re: Anyone know anything about string encodings? Arne Vajhøj <arne@vajhoej.dk> - 2017-07-25 09:16 -0400
                  Re: Anyone know anything about string encodings? Jukka Lahtinen <jtfjdehf@hotmail.com.invalid> - 2017-07-25 11:50 +0300
                    Re: Anyone know anything about string encodings? Jukka Lahtinen <jtfjdehf@hotmail.com.invalid> - 2017-07-25 11:51 +0300
                      Re: Anyone know anything about string encodings? Joshua Cranmer 🐧 <Pidgeot18@verizon.invalid> - 2017-07-25 09:39 -0500
                        Re: Anyone know anything about string encodings? Arne Vajhøj <arne@vajhoej.dk> - 2017-07-25 14:34 -0400
                    Re: Anyone know anything about string encodings? "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> - 2017-07-25 13:07 +0200
                    Re: Anyone know anything about string encodings? Arne Vajhøj <arne@vajhoej.dk> - 2017-07-25 09:12 -0400
                      Re: Anyone know anything about string encodings? Jukka Lahtinen <jtfjdehf@hotmail.com.invalid> - 2017-07-25 18:05 +0300
                        Re: Anyone know anything about string encodings? Arne Vajhøj <arne@vajhoej.dk> - 2017-07-25 14:31 -0400
                      Re: Anyone know anything about string encodings? lewbloch@gmail.com - 2017-07-26 11:23 -0700
    Re: Anyone know anything about string encodings? Eric Sosman <esosman@comcast-dot-net.invalid> - 2017-07-20 18:09 -0400
      Re: Anyone know anything about string encodings? Eric Douglas <e.d.programmer@gmail.com> - 2017-07-21 06:09 -0700
        Re: Anyone know anything about string encodings? Arne Vajhøj <arne@vajhoej.dk> - 2017-07-21 19:46 -0400
    Re: Anyone know anything about string encodings? Arne Vajhøj <arne@vajhoej.dk> - 2017-07-20 20:18 -0400

Page 1 of 2  [1] 2  Next page →

#36970 — Anyone know anything about string encodings?

I get an internet stream, in this case a signature from the G Suite Google feed.
It returns from this method:
org.apache.http.HttpResponse.getEntity().getContent()

Then I try to pass it into org.apache.commons.io.IOUtils.toString().
This method accepts a stream and a charset to return a String.

I tried to pass it a null and it took the default, which on our system came up with "windows-1252", and the new signatures looked wrong.
I passed in "UTF-8" and they look right, at least for now.  Will this always work, or how would I choose what charset it should convert with?

[toc] | [next] | [standalone]

#36971

On 7/20/2017 3:22 PM, Eric Douglas wrote:
> I get an internet stream, in this case a signature from the G Suite
> Google feed. It returns from this method: 
> org.apache.http.HttpResponse.getEntity().getContent()
> 
> Then I try to pass it into org.apache.commons.io.IOUtils.toString(). 
> This method accepts a stream and a charset to return a String.
> 
> I tried to pass it a null and it took the default, which on our
> system came up with "windows-1252", and the new signatures looked
> wrong. I passed in "UTF-8" and they look right, at least for now.
> Will this always work, or how would I choose what charset it should
> convert with?

*Breathes in after a spate of uncontrollable laughter.*

It looks like you're getting the response body of an HTTP response. In 
practice, the Content-Type of HTTP headers tends to be woefully 
unreliable, and the charset parameter is often in center field of 
miscommunication here. If you're working with HTML data, then the 
general rule boils down to:

1. Use the charset field within the document (if present)
2. Else, use the Content-Type header (if present)
3. Otherwise, play the game of charset roulette. 
UTF-8-if-valid-else-Windows-1252 is usually the easiest way to win this 
game.

-- 
Beware of bugs in the above code; I have only proved it correct, not
tried it. -- Donald E. Knuth

[toc] | [prev] | [next] | [standalone]

#36974

On Thursday, July 20, 2017 at 6:04:12 PM UTC-4, Joshua Cranmer 🐧 wrote:
> 
> 1. Use the charset field within the document (if present)
> 2. Else, use the Content-Type header (if present)
> 3. Otherwise, play the game of charset roulette. 
> UTF-8-if-valid-else-Windows-1252 is usually the easiest way to win this 
> game.
> 

Well, I see Google put it in the parameters.  Is this good?

          String charset = null;
          final HttpEntity ent = httpResp.getEntity();
          for (Header hdr : httpResp.getAllHeaders()) {
               for (HeaderElement he : hdr.getElements()) {
                    for (NameValuePair nvp : he.getParameters()) {
                         if (nvp.getName().equals("charset")) {
                              charset = nvp.getValue();
                              break;
                         }
                    }
                    if (charset != null) break;
               }
               if (charset != null) break;
          }

[toc] | [prev] | [next] | [standalone]

#36978

On 7/20/2017 6:03 PM, Joshua Cranmer 🐧 wrote:
> On 7/20/2017 3:22 PM, Eric Douglas wrote:
>> I get an internet stream, in this case a signature from the G Suite
>> Google feed. It returns from this method: 
>> org.apache.http.HttpResponse.getEntity().getContent()
>>
>> Then I try to pass it into org.apache.commons.io.IOUtils.toString(). 
>> This method accepts a stream and a charset to return a String.
>>
>> I tried to pass it a null and it took the default, which on our
>> system came up with "windows-1252", and the new signatures looked
>> wrong. I passed in "UTF-8" and they look right, at least for now.
>> Will this always work, or how would I choose what charset it should
>> convert with?

> It looks like you're getting the response body of an HTTP response. In 
> practice, the Content-Type of HTTP headers tends to be woefully 
> unreliable, and the charset parameter is often in center field of 
> miscommunication here. If you're working with HTML data, then the 
> general rule boils down to:
> 
> 1. Use the charset field within the document (if present)
> 2. Else, use the Content-Type header (if present)
> 3. Otherwise, play the game of charset roulette. 
> UTF-8-if-valid-else-Windows-1252 is usually the easiest way to win this 
> game.

I think my suggested algorithm handle your case 3 more
according to standards.

Arne

[toc] | [prev] | [next] | [standalone]

#36980

On 7/21/2017 6:47 PM, Arne Vajhøj wrote:
> I think my suggested algorithm handle your case 3 more
> according to standards.

For HTML, the standard more or less says "if there's no charset from the 
document itself or from the Content-Type, do whatever you want to."

In practice, web browsers do different things. I believe IE tries to do 
autodetection. I know Firefox falls back onto a locale-dependent default 
encoding (e.g., en-US defaults to Windows-1252, but ja-JP would use 
Shift-JIS). I don't know about Chrome or Safari.

Testing for UTF-8 is pretty much a safe first step because a) everyone 
already recommends using UTF-8 as the default anyways and b) it's very 
distinctive in its output format to the point that virtually all 8-bit 
non-UTF-8 text will fail the UTF-8 parsing. ICU, e.g., will say that 
text is UTF-8 with 50% confidence if it sees just 1 UTF-8 sequence and 
no invalid sequences moving to 100% confidence upon the third valid 
sequence.
> 
> Arne
> 
> 


-- 
Beware of bugs in the above code; I have only proved it correct, not 
tried it. -- Donald E. Knuth

[toc] | [prev] | [next] | [standalone]

#36981

On 7/22/2017 1:20 PM, Joshua Cranmer 🐧 wrote:
> On 7/21/2017 6:47 PM, Arne Vajhøj wrote:
>> I think my suggested algorithm handle your case 3 more
>> according to standards.
> 
> For HTML, the standard more or less says "if there's no charset from the 
> document itself or from the Content-Type, do whatever you want to."
> 
> In practice, web browsers do different things. I believe IE tries to do 
> autodetection. I know Firefox falls back onto a locale-dependent default 
> encoding (e.g., en-US defaults to Windows-1252, but ja-JP would use 
> Shift-JIS). I don't know about Chrome or Safari.
> 
> Testing for UTF-8 is pretty much a safe first step because a) everyone 
> already recommends using UTF-8 as the default anyways and b) it's very 
> distinctive in its output format to the point that virtually all 8-bit 
> non-UTF-8 text will fail the UTF-8 parsing. ICU, e.g., will say that 
> text is UTF-8 with 50% confidence if it sees just 1 UTF-8 sequence and 
> no invalid sequences moving to 100% confidence upon the third valid 
> sequence.

ISO-8859-1 used to be the default for HTML pages.

But checking the HTML 5 specs I can not find it. It does say
that the default encoding for eastern Europe is ISO-8859-2,
windows-1251 (which is close to ISO-8859-5) etc.. But I can not
find a place where it specify ISO-8859-1 (or windows-1252)
to be default for western Europe.

????

Arne

[toc] | [prev] | [next] | [standalone]

#36984

On 7/22/2017 3:40 PM, Arne Vajhøj wrote:
> But checking the HTML 5 specs I can not find it. It does say
> that the default encoding for eastern Europe is ISO-8859-2,
> windows-1251 (which is close to ISO-8859-5) etc.. But I can not
> find a place where it specify ISO-8859-1 (or windows-1252)
> to be default for western Europe.

HTML 5 section 8.2.2.2, item 8. Or section 12.2.3.2 if you're following 
WHATWG's version.

-- 
Beware of bugs in the above code; I have only proved it correct, not 
tried it. -- Donald E. Knuth

[toc] | [prev] | [next] | [standalone]

#36985

On 7/24/2017 10:27 AM, Joshua Cranmer 🐧 wrote:
> On 7/22/2017 3:40 PM, Arne Vajhøj wrote:
>> But checking the HTML 5 specs I can not find it. It does say
>> that the default encoding for eastern Europe is ISO-8859-2,
>> windows-1251 (which is close to ISO-8859-5) etc.. But I can not
>> find a place where it specify ISO-8859-1 (or windows-1252)
>> to be default for western Europe.
> 
> HTML 5 section 8.2.2.2, item 8. Or section 12.2.3.2 if you're following 
> WHATWG's version.

Yes. But it was item 9 where I wondered why no western europe.

Arne

[toc] | [prev] | [next] | [standalone]

#36986

On 7/24/2017 9:54 AM, Arne Vajhøj wrote:
> Yes. But it was item 9 where I wondered why no western europe.

The default for anything not listed is windows-1252.

-- 
Beware of bugs in the above code; I have only proved it correct, not 
tried it. -- Donald E. Knuth

[toc] | [prev] | [next] | [standalone]

#36987

Except on Linux or Macs, no?

[toc] | [prev] | [next] | [standalone]

#36994

On 7/24/2017 8:40 PM, lewbloch@gmail.com wrote:
> Except on Linux or Macs, no?

That does not really matter. Except that they may not have
windows-1252 and may have to settle for ISO-8859-1 as being
compatible enough.

Remember the context:

HTML producer on PC using 20+ year old SW--->server--->browser on PC

The fact that the browser PC is running on Linux or MacOS X does
(probably) not impact the likelihood that the producer PC runs
Windows.

Arne

[toc] | [prev] | [next] | [standalone]

#36989

Joshua Cranmer 🐧 <Pidgeot18@verizon.invalid> writes:
> On 7/24/2017 9:54 AM, Arne Vajhøj wrote:

>> Yes. But it was item 9 where I wondered why no western europe.
> The default for anything not listed is windows-1252.

Anything windows-specific should NEVER be used as a default.
Only vendor-independent standards should ever be default!

-- 
Jukka Lahtinen

[toc] | [prev] | [next] | [standalone]

#36990

Jukka Lahtinen <jtfjdehf@hotmail.com.invalid> writes:
> Joshua Cranmer 🐧 <Pidgeot18@verizon.invalid> writes:
>> On 7/24/2017 9:54 AM, Arne Vajhøj wrote:
>
>>> Yes. But it was item 9 where I wondered why no western europe.
>> The default for anything not listed is windows-1252.
>
> Anything windows-specific should NEVER be used as a default.
> Only vendor-independent standards should ever be default!

And in the case of character sets, ISO-8859-1 or ISO-8859-15.

-- 
Jukka Lahtinen

[toc] | [prev] | [next] | [standalone]

#36995

On 7/25/2017 3:51 AM, Jukka Lahtinen wrote:
> Jukka Lahtinen <jtfjdehf@hotmail.com.invalid> writes:
>> Joshua Cranmer 🐧 <Pidgeot18@verizon.invalid> writes:
>>> On 7/24/2017 9:54 AM, Arne Vajhøj wrote:
>>
>>>> Yes. But it was item 9 where I wondered why no western europe.
>>> The default for anything not listed is windows-1252.
>>
>> Anything windows-specific should NEVER be used as a default.
>> Only vendor-independent standards should ever be default!
> 
> And in the case of character sets, ISO-8859-1 or ISO-8859-15.

If you ever have a choice, choose UTF-8. Do not pick ISO-8859-* 
charsets, particularly since you stand a good chance of mislabelling 
them. (Much text labelled ISO-8859-1 is actually Windows-1252. What's 
the difference? ISO-8859-1 forbids the C1 area, Windows-1252 fills that 
with some extra characters).

-- 
Beware of bugs in the above code; I have only proved it correct, not 
tried it. -- Donald E. Knuth

[toc] | [prev] | [next] | [standalone]

#36998

On 7/25/2017 10:39 AM, Joshua Cranmer 🐧 wrote:
> On 7/25/2017 3:51 AM, Jukka Lahtinen wrote:
>> Jukka Lahtinen <jtfjdehf@hotmail.com.invalid> writes:
>>> Joshua Cranmer 🐧 <Pidgeot18@verizon.invalid> writes:
>>>> On 7/24/2017 9:54 AM, Arne Vajhøj wrote:
>>>
>>>>> Yes. But it was item 9 where I wondered why no western europe.
>>>> The default for anything not listed is windows-1252.
>>>
>>> Anything windows-specific should NEVER be used as a default.
>>> Only vendor-independent standards should ever be default!
>>
>> And in the case of character sets, ISO-8859-1 or ISO-8859-15.
> 
> If you ever have a choice, choose UTF-8. Do not pick ISO-8859-* 
> charsets, particularly since you stand a good chance of mislabelling 
> them.

Not so sure.

UTF-8 is common today. ISO-8859-1 was common many years ago.

Today it is common to specify charset. Many years ago it was
not common to specify charset.

I think the probability of unspecified being ISO-8859-1
is pretty high.

And ISO-8859-1 interoreted as UTF-8 does not look good.

> (Much text labelled ISO-8859-1 is actually Windows-1252. What's 
> the difference? ISO-8859-1 forbids the C1 area, Windows-1252 fills that 
> with some extra characters).

I doubt that those are that widely used.

Arne

[toc] | [prev] | [next] | [standalone]

#36991

On 25.07.2017 10:50, Jukka Lahtinen wrote:
> Joshua Cranmer 🐧 <Pidgeot18@verizon.invalid> writes:
>> On 7/24/2017 9:54 AM, Arne Vajhøj wrote:
> 
>>> Yes. But it was item 9 where I wondered why no western europe.
>> The default for anything not listed is windows-1252.
> 
> Anything windows-specific should NEVER be used as a default.
> Only vendor-independent standards should ever be default!
> 

Be lenient in what data you accept, strict in what you give out.


Cheers!,

- Alf

[toc] | [prev] | [next] | [standalone]

#36993

On 7/25/2017 4:50 AM, Jukka Lahtinen wrote:
> Joshua Cranmer 🐧 <Pidgeot18@verizon.invalid> writes:
>> On 7/24/2017 9:54 AM, Arne Vajhøj wrote:
>>> Yes. But it was item 9 where I wondered why no western europe.
>> The default for anything not listed is windows-1252.
> 
> Anything windows-specific should NEVER be used as a default.
> Only vendor-independent standards should ever be default!

The difference between windows-1252 and ISO-8859-1 should
be negligible.

Arne

[toc] | [prev] | [next] | [standalone]

#36996

Arne Vajhøj <arne@vajhoej.dk> writes:
> On 7/25/2017 4:50 AM, Jukka Lahtinen wrote:
>> Joshua Cranmer 🐧 <Pidgeot18@verizon.invalid> writes:
>>> On 7/24/2017 9:54 AM, Arne Vajhøj wrote:
>>>> Yes. But it was item 9 where I wondered why no western europe.
>>> The default for anything not listed is windows-1252.

>> Anything windows-specific should NEVER be used as a default.
>> Only vendor-independent standards should ever be default!

> The difference between windows-1252 and ISO-8859-1 should
> be negligible.

The main difference in many cases being that windows-1252 is
Microsoft-specific and ISO-8859-1 is a universal standard.

-- 
Jukka Lahtinen

[toc] | [prev] | [next] | [standalone]

#36997

On 7/25/2017 11:05 AM, Jukka Lahtinen wrote:
> Arne Vajhøj <arne@vajhoej.dk> writes:
>> On 7/25/2017 4:50 AM, Jukka Lahtinen wrote:
>>> Joshua Cranmer 🐧 <Pidgeot18@verizon.invalid> writes:
>>>> On 7/24/2017 9:54 AM, Arne Vajhøj wrote:
>>>>> Yes. But it was item 9 where I wondered why no western europe.
>>>> The default for anything not listed is windows-1252.
> 
>>> Anything windows-specific should NEVER be used as a default.
>>> Only vendor-independent standards should ever be default!
> 
>> The difference between windows-1252 and ISO-8859-1 should
>> be negligible.
> 
> The main difference in many cases being that windows-1252 is
> Microsoft-specific and ISO-8859-1 is a universal standard.

There is a huge difference from standard perspective.

My comment about the difference being negligible was
only for the actual rendering.

Arne

[toc] | [prev] | [next] | [standalone]

#37000

On Tuesday, July 25, 2017 at 6:12:39 AM UTC-7, Arne Vajhøj wrote:
> On 7/25/2017 4:50 AM, Jukka Lahtinen wrote:
> > Joshua Cranmer 🐧 writes:
> >> On 7/24/2017 9:54 AM, Arne Vajhøj wrote:
> >>> Yes. But it was item 9 where I wondered why no western europe.
> >> The default for anything not listed is windows-1252.
> > 
> > Anything windows-specific should NEVER be used as a default.
> > Only vendor-independent standards should ever be default!
> 
> The difference between windows-1252 and ISO-8859-1 should
> be negligible.

"Should be" != "is".  I've seen glitches between those two encodings.

-- 
Lew

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.java.programmer

csiph-web