Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #167714 > unrolled thread

Iceweasel uses wrong charset

Back to article view | Back to linux.debian.user

Contents

  Iceweasel uses wrong charset Rob van der Putten <rob@sput.nl> - 2016-05-11 11:00 +0200
    Re: Iceweasel uses wrong charset Frédéric Marchal <frederic.marchal@wowtechnology.com> - 2016-05-11 11:20 +0200
    Re: Iceweasel uses wrong charset Michael Lange <klappnase@freenet.de> - 2016-05-11 11:20 +0200
      Re: Iceweasel uses wrong charset err404@free.fr - 2016-05-11 11:30 +0200
        Re: Iceweasel uses wrong charset Frédéric Marchal <frederic.marchal@wowtechnology.com> - 2016-05-11 12:00 +0200
          Re: Iceweasel uses wrong charset Vincent Lefevre <vincent@vinc17.net> - 2016-05-13 12:10 +0200
      Re: Iceweasel uses wrong charset Rob van der Putten <rob@sput.nl> - 2016-05-11 12:00 +0200
        Re: Iceweasel uses wrong charset <tomas@tuxteam.de> - 2016-05-11 12:20 +0200
          Re: Iceweasel uses wrong charset Rob van der Putten <rob@sput.nl> - 2016-05-11 12:40 +0200
            Re: Iceweasel uses wrong charset <tomas@tuxteam.de> - 2016-05-11 12:50 +0200
              Re: Iceweasel uses wrong charset Rob van der Putten <rob@sput.nl> - 2016-05-11 15:20 +0200
                Re: Iceweasel uses wrong charset <tomas@tuxteam.de> - 2016-05-11 16:30 +0200
                  Re: Iceweasel uses wrong charset Michael Lange <klappnase@freenet.de> - 2016-05-11 18:50 +0200
                    [partially solved]Re: Iceweasel uses wrong charset Michael Lange <klappnase@freenet.de> - 2016-05-11 19:00 +0200
                      Re: [partially solved]Re: Iceweasel uses wrong charset <tomas@tuxteam.de> - 2016-05-12 08:30 +0200
                      Re: [partially solved]Re: Iceweasel uses wrong charset Rob van der Putten <rob@sput.nl> - 2016-05-12 18:00 +0200
          Re: Iceweasel uses wrong charset Vincent Lefevre <vincent@vinc17.net> - 2016-05-13 12:20 +0200
        Re: Iceweasel uses wrong charset Richard Owlett <rowlett@cloud85.net> - 2016-05-11 12:30 +0200

#167714 — Iceweasel uses wrong charset

Hi there


UTF-8 is my default locale. So obviously text files are UTF-8 as well.
So why does iceweasel insist that these are windows-1252? How do I fix this?


Regards,
Rob

[toc] | [next] | [standalone]

#167716

On Wednesday 11 May 2016 10:55:54 Rob van der Putten wrote:
> Hi there
> 
> 
> UTF-8 is my default locale. So obviously text files are UTF-8 as well.
> So why does iceweasel insist that these are windows-1252? How do I fix this?

In about:config, have you changed any of intl.charset.detector or 
intl.charset.fallback.override ? They are both empty in my configuration.

Frederic

[toc] | [prev] | [next] | [standalone]

#167717

Hi,

On Wed, 11 May 2016 10:55:54 +0200
Rob van der Putten <rob@sput.nl> wrote:

> Hi there
> 
> 
> UTF-8 is my default locale. So obviously text files are UTF-8 as well.
> So why does iceweasel insist that these are windows-1252? How do I fix
> this?

you can try the following:

type "about:config" into iceweasel's address bar (without the quotes of
course), then type "charset" into the config's search bar. Several options
should appear. Look for the key "intl.charset.default". Is the value
already set to "UTF-8"? If not, double-click the current value and change
it to UTF-8 , this should hopefully fix it.

Best regards

Michael


.-.. .. ...- .   .-.. --- -. --.   .- -. -..   .--. .-. --- ... .--. . .-.

You humans have that emotional need to express gratitude.  "You're
welcome," I believe, is the correct response.
		-- Spock, "Bread and Circuses", stardate 4041.2

[toc] | [prev] | [next] | [standalone]

#167718

[Multipart message — attachments visible in raw view] — view raw

Why unicode like utf-8 is not default value in iceweasel ?????

(it's time, we are in 2016 now)

[toc] | [prev] | [next] | [standalone]

#167719

On Wednesday 11 May 2016 11:23:56 err404@free.fr wrote:
> Why unicode like utf-8 is not default value in iceweasel ?????
> 
> (it's time, we are in 2016 now)

Unicode is the default in firefox/iceweasel. See 
https://developer.mozilla.org/en-US/docs/Web/Guide/Localizations_and_character_encodings

The OP may have changed an option in about:config to make windows-1252 the 
default.

Frederic

[toc] | [prev] | [next] | [standalone]

#167804

On 2016-05-11 11:55:33 +0200, Frédéric Marchal wrote:
> On Wednesday 11 May 2016 11:23:56 err404@free.fr wrote:
> > Why unicode like utf-8 is not default value in iceweasel ?????
> > 
> > (it's time, we are in 2016 now)
> 
> Unicode is the default in firefox/iceweasel.

No, the default is windows-1252 (checked by testing with a fresh
profile).

> See 
> https://developer.mozilla.org/en-US/docs/Web/Guide/Localizations_and_character_encodings

which says that the default is windows-1252.

BTW, this page says: "New-authored locale-native UTF-8 content is
expected to declare its encoding, in which case the fallback encoding
does not participate in the processing of content." but I don't see
how one can declare the encoding for text files read locally with the
"file:" URL scheme.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

[toc] | [prev] | [next] | [standalone]

#167720

Hi there


Michael Lange wrote:

> you can try the following:
>
> type "about:config" into iceweasel's address bar (without the quotes of
> course), then type "charset" into the config's search bar. Several options
> should appear. Look for the key "intl.charset.default". Is the value
> already set to "UTF-8"? If not, double-click the current value and change
> it to UTF-8 , this should hopefully fix it.

This seems the obvious thing to do.
It doens't work though.

Regards,
Rob

[toc] | [prev] | [next] | [standalone]

#167721

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, May 11, 2016 at 11:58:31AM +0200, Rob van der Putten wrote:
> Hi there
> 
> 
> Michael Lange wrote:
> 
> >you can try the following:
> >
> >type "about:config" into iceweasel's address bar (without the quotes of
> >course), then type "charset" into the config's search bar. Several options
> >should appear. Look for the key "intl.charset.default". Is the value
> >already set to "UTF-8"? If not, double-click the current value and change
> >it to UTF-8 , this should hopefully fix it.
> 
> This seems the obvious thing to do.
> It doens't work though.

This is happening only on local files? Perhaps FF is expecting that brain
damaged byte order mark to recognize the file as UTF-8?

When the data come over HTTP there are other out-of-band hints to communicate
the encoding.

Try editing the file with some Microsoft cra^H^H^H text editor and saving
as "Unicode": that might insert the BOM at the beginning.

Just a shot in the dark.

regards
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlczBRAACgkQBcgs9XrR2kZp1QCfX9xv/KDqXjkuz7T0wzJYJf+j
a8EAn1GrBIEywFRyb0ypRnJPsvQKNdtP
=7HZt
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#167725

Hi there


tomas@tuxteam.de wrote:

> This is happening only on local files? Perhaps FF is expecting that brain
> damaged byte order mark to recognize the file as UTF-8?
>
> When the data come over HTTP there are other out-of-band hints to communicate
> the encoding.
>
> Try editing the file with some Microsoft cra^H^H^H text editor and saving
> as "Unicode": that might insert the BOM at the beginning.
>
> Just a shot in the dark.

I actually thought about this. But it just seems crude;

vim some.txt
^vufeff<Enter>
<Esc>:wq<Enter>

It does work though.
But I want UTF-8 without a BOM. It is, after all, the default locale.
Is Firefox really this braindead?

Note: If you edit this again, vim doesn't show the BOM any more. It's 
there though.


Regards,
Rob

[toc] | [prev] | [next] | [standalone]

#167726

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, May 11, 2016 at 12:34:29PM +0200, Rob van der Putten wrote:
> Hi there
> 
> 
> tomas@tuxteam.de wrote:

[BOM BOM hurrah!]

> I actually thought about this. But it just seems crude;
> 
> vim some.txt
> ^vufeff<Enter>
> <Esc>:wq<Enter>
> 
> It does work though.
> But I want UTF-8 without a BOM. It is, after all, the default locale.

Everyone wants that... except Microsoft, it seems.

> Is Firefox really this braindead?

Sometimes I fear that too.

> Note: If you edit this again, vim doesn't show the BOM any more.
> It's there though.

That's because it has been enshrined by the Unicode Consortium. I think
I know who pushed that (Apple might have helped, though). So Vim's
correct in tolerating that on input.

Perhaps you can help yourself by even uglier tricks (like with an
http-equiv header in your HTML). But best would be if you could
exorcise this abject behaviour from Firefox.

Let us know...

regards
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlczC/wACgkQBcgs9XrR2kZVDwCfYu0BnNtQIxXoh39Jq4UCWQl/
nyEAnifLHZX0KAUL75j1RFApbp/DOqry
=JOlh
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#167729

Hi there


tomas@tuxteam.de wrote:

> On Wed, May 11, 2016 at 12:34:29PM +0200, Rob van der Putten wrote:

<Cut>

>> I actually thought about this. But it just seems crude;
>>
>> vim some.txt
>> ^vufeff<Enter>
>> <Esc>:wq<Enter>
>>
>> It does work though.
>> But I want UTF-8 without a BOM. It is, after all, the default locale.
>
> Everyone wants that... except Microsoft, it seems.
>
>> Is Firefox really this braindead?
>
> Sometimes I fear that too.
>
>> Note: If you edit this again, vim doesn't show the BOM any more.
>> It's there though.
>
> That's because it has been enshrined by the Unicode Consortium. I think
> I know who pushed that (Apple might have helped, though). So Vim's
> correct in tolerating that on input.

I don't think they like BOM. After all they named it 'ZERO WIDTH 
NO-BREAK SPACE'.

> Perhaps you can help yourself by even uglier tricks (like with an
> http-equiv header in your HTML). But best would be if you could
> exorcise this abject behaviour from Firefox.

For HTML this works.

I can even put hieroglyphs in a plain text file and print it using a 
plain text print queue. UTF-8 just works.
Firefox however, insists on putting a square peg into a round hole.

> Let us know...

The only way to fix this is probably to edit the source.

I have a very old iceape that actually gets it right. An older version 
had a 'remove all bookmarks' button, which no longer exists.
With iceweazel one actually has to create a bookmark to switch 
javascript on and of (the plug-in for this does not work).
And with jesie's apache there is no longer a default mime-type or 
cern-meta module because _NO_ _ONE_ uses this any more.
Apparently I'm a figment of my own imagination.


Regards,
rob

[toc] | [prev] | [next] | [standalone]

#167731

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, May 11, 2016 at 03:14:39PM +0200, Rob van der Putten wrote:
> Hi there
> 
> 
> tomas@tuxteam.de wrote:
> 
> >On Wed, May 11, 2016 at 12:34:29PM +0200, Rob van der Putten wrote:

[...]

> >That's because it has been enshrined by the Unicode Consortium [...]

> I don't think they like BOM. After all they named it 'ZERO WIDTH
> NO-BREAK SPACE'.

[chuckle] Yeah. But they should have just thrown it out. It's doing
damage for no appreciable win. Corporations should have to pay back
for the collateral they produce by pushing their agenda.

[...]

> The only way to fix this is probably to edit the source.

There's some hope. Have a look at the intl subcategory in about:config[1]
(especially at intl.locale.matchOS and/or general.useragent.locale),
but also intl.accept.charsets (it is a list, might be prioritized).

I don't feel like poking in the browser mud at the moment -- I'm
deeply in need of something more rewarding, sorry :-/

> I have a very old iceape that actually gets it right. An older
> version had a 'remove all bookmarks' button, which no longer exists.
> With iceweazel one actually has to create a bookmark to switch
> javascript on and of (the plug-in for this does not work).
> And with jesie's apache there is no longer a default mime-type or
> cern-meta module because _NO_ _ONE_ uses this any more.
> Apparently I'm a figment of my own imagination.

You don't exist. If that's any consolation, I don't exist either.

[1] http://kb.mozillazine.org/Firefox_:_FAQs_:_About:config_Entries#Intl.

hth
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlczQXMACgkQBcgs9XrR2ka3kACeKHSFOnb/8V2MAd8uTJqyFFjM
iNYAnAtj97lOSxnD11RLv4pGa5I8RjZT
=wfKw
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#167736

On Wed, 11 May 2016 16:28:03 +0200
<tomas@tuxteam.de> wrote:

> There's some hope. Have a look at the intl subcategory in about:config
> [1] (especially at intl.locale.matchOS and/or general.useragent.locale),
> but also intl.accept.charsets (it is a list, might be prioritized).
> 

I never tested this before, but actually I have the same problem with
UTF-8 files here as the OP described. intl.locale.matchOS doesn't help any
here, it was already set to true.
According to this (yet probably dated ) page:
   http://www-archive.mozilla.org/projects/intl/chardet.html
universal_charset_detector might help if it were enabled by default.

Regards

Michael

.-.. .. ...- .   .-.. --- -. --.   .- -. -..   .--. .-. --- ... .--. . .-.

Another Armenia, Belgium ... the weak innocents who always seem to be
located on a natural invasion route.
		-- Kirk, "Errand of Mercy", stardate 3198.4

[toc] | [prev] | [next] | [standalone]

#167739 — [partially solved]Re: Iceweasel uses wrong charset

On Wed, 11 May 2016 18:43:50 +0200
Michael Lange <klappnase@freenet.de> wrote:

> According to this (yet probably dated ) page:
>    http://www-archive.mozilla.org/projects/intl/chardet.html
> universal_charset_detector might help if it were enabled by default.

Just looked at the above page again, and noticed that the "japanese"
charset detector also supports UTF-8 and cp1252. So I tried to set
intl.charset.detector to "ja_parallel_state_machine" and actually at least
the minimal utf-8 text file I just tested (which contains no more than a
bunch of german umlaut characters) is now displayed correctly.

Regards

Michael


.-.. .. ...- .   .-.. --- -. --.   .- -. -..   .--. .-. --- ... .--. . .-.

	"Life and death are seldom logical."
	"But attaining a desired goal always is."
		-- McCoy and Spock, "The Galileo Seven", stardate 2821.7

[toc] | [prev] | [next] | [standalone]

#167760 — Re: [partially solved]Re: Iceweasel uses wrong charset

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, May 11, 2016 at 06:55:31PM +0200, Michael Lange wrote:
> On Wed, 11 May 2016 18:43:50 +0200
> Michael Lange <klappnase@freenet.de> wrote:
> 
> > According to this (yet probably dated ) page:
> >    http://www-archive.mozilla.org/projects/intl/chardet.html
> > universal_charset_detector might help if it were enabled by default.
> 
> Just looked at the above page again, and noticed that the "japanese"
> charset detector also supports UTF-8 and cp1252. So I tried to set
> intl.charset.detector to "ja_parallel_state_machine" and actually at least
> the minimal utf-8 text file I just tested (which contains no more than a
> bunch of german umlaut characters) is now displayed correctly.

This is all pretty unfortunate: we're replacing simple, understandable
mechanisms ("hey, Firefox: on my computer all text files are supposed
to be utf-8, promised") by complex heuristics (which, as a very last
resort are cool, mind you: "dunno what that file is: try to make a guess,
please"), but *not* as the up-front decision procedure. Besides, who
knows what vulnerabilities lurk in there?

Hey, Mozilla! If you are listening (do you listen at all?) Where is
your user's sovereignity?

- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlc0IhMACgkQBcgs9XrR2kYSkwCeNVxGPMtBPPOzYcOhrCqk6otf
cd0An0C+T9DLagA27QP5yCSxbKKpAAKC
=QSts
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#167777 — Re: [partially solved]Re: Iceweasel uses wrong charset

Hi there


Michael Lange wrote:

> On Wed, 11 May 2016 18:43:50 +0200
> Michael Lange<klappnase@freenet.de>  wrote:
>
>> According to this (yet probably dated ) page:
>>     http://www-archive.mozilla.org/projects/intl/chardet.html
>> universal_charset_detector might help if it were enabled by default.
>
> Just looked at the above page again, and noticed that the "japanese"
> charset detector also supports UTF-8 and cp1252. So I tried to set
> intl.charset.detector to "ja_parallel_state_machine" and actually at least
> the minimal utf-8 text file I just tested (which contains no more than a
> bunch of german umlaut characters) is now displayed correctly.

Yesterday I found https://bugzilla.mozilla.org/show_bug.cgi?id=910192
But gmane dropped my post.


Regards,
Rob

[toc] | [prev] | [next] | [standalone]

#167805

On 2016-05-11 12:10:25 +0200, tomas@tuxteam.de wrote:
> This is happening only on local files? Perhaps FF is expecting that
> brain damaged byte order mark to recognize the file as UTF-8?

I reported a bug about that in the past:

  https://bugzilla.mozilla.org/show_bug.cgi?id=1071816

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

[toc] | [prev] | [next] | [standalone]

#167722

On 5/11/2016 4:58 AM, Rob van der Putten wrote:
> Hi there
>
>
> Michael Lange wrote:
>
>> you can try the following:
>>
>> type "about:config" into iceweasel's address bar (without the
>> quotes of
>> course), then type "charset" into the config's search bar.
>> Several options
>> should appear. Look for the key "intl.charset.default". Is the
>> value
>> already set to "UTF-8"? If not, double-click the current value
>> and change
>> it to UTF-8 , this should hopefully fix it.
>
> This seems the obvious thing to do.
> It doens't work though.
>

http://kb.mozillazine.org/Firefox_:_FAQs_:_About:config_Entries#Intl.

The trailing "."   *IS*   part ot the URL

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web