Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #35455

Re: how to detect the character encoding in a web page ?

Path csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!ecngs!feeder2.ecngs.de!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path <kwpolska@gmail.com>
X-Original-To python-list@python.org
Delivered-To python-list@mail.python.org
X-Spam-Status OK 0.038
X-Spam-Evidence '*H*': 0.92; '*S*': 0.00; 'charset': 0.09; 'definition,': 0.09; 'useless': 0.09; 'kurt': 0.11; 'dec': 0.15; '24,': 0.16; 'gpg': 0.16; 'tags.': 0.16; 'mon,': 0.16; 'wrote:': 0.17; 'detect': 0.17; 'subject:page': 0.17; 'somebody': 0.23; 'second': 0.24; 'header:In-Reply-To:1': 0.25; 'am,': 0.27; 'websites,': 0.27; 'message-id:@mail.gmail.com': 0.27; 'subject: ?': 0.30; 'url:python': 0.32; 'to:addr:python-list': 0.33; 'received:google.com': 0.34; 'received:209.85': 0.35; 'something': 0.35; 'but': 0.36; 'url:org': 0.36; 'uses': 0.37; 'received:209': 0.37; 'subject:: ': 0.38; 'to:addr:python.org': 0.39; 'header:Received:5': 0.40; 'websites': 0.66; 'today.': 0.69; 'guaranteed': 0.76; 'html5,': 0.84; 'today\xe2\x80\x99s': 0.84; 'url:tk': 0.93; 'confidence': 0.95
DKIM-Signature v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=q9K2e8vuopwOfzpJvmiNPs++l3/EaEAzeJENirjBeCY=; b=TkgapCdd96vkncAEe+NhZN6giPEhgqnctLs38oh4qITlSGP1Sk1MKGJ6ABRpPUsYxp m1vAh4zvf8J0VqwO54L74ortON+yR3vb0nXb2AG16LlLflHG8Sxr4LI7+DOYGuhP3mgV 9fKyKH7NAwD5zSsCHc+S6bVamnMa37ItNv98rfenwkLCe4UYQ6qyj7MBENq1/7xxLPWG ary+48jU5yjZOTCWe0mWGJ3FdT+4P9j4cW+NbexkMLNTe5XNQ+51rZ4G1T0xIRyBO0pw y6wYKaW3RuBVgQVQyBQ/vcsicZPWyYkNl6HkLI5YoDUT8xdV26Z/stNcRMvlr6Cv9vfZ 1HmA==
MIME-Version 1.0
In-Reply-To <5C06B25F-066B-421E-9849-2E1B2EAFFEBE@gmail.com>
References <c15bad9a-a7f7-456e-8dc5-b1af67fbdd44@googlegroups.com> <2324928c-32de-4f9d-8ff1-5db6dcf5543a@googlegroups.com> <5C06B25F-066B-421E-9849-2E1B2EAFFEBE@gmail.com>
Date Mon, 24 Dec 2012 13:16:16 +0100
Subject Re: how to detect the character encoding in a web page ?
From Kwpolska <kwpolska@gmail.com>
To python-list@python.org
Content-Type text/plain; charset=UTF-8
Content-Transfer-Encoding quoted-printable
X-BeenThere python-list@python.org
X-Mailman-Version 2.1.15
Precedence list
List-Id General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive <http://mail.python.org/pipermail/python-list/>
List-Post <mailto:python-list@python.org>
List-Help <mailto:python-list-request@python.org?subject=help>
List-Subscribe <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups comp.lang.python
Message-ID <mailman.1253.1356351379.29569.python-list@python.org> (permalink)
Lines 30
NNTP-Posting-Host 2001:888:2000:d::a6
X-Trace 1356351379 news.xs4all.nl 6939 [2001:888:2000:d::a6]:49457
X-Complaints-To abuse@xs4all.nl
Xref csiph.com comp.lang.python:35455

Show key headers only | View raw

On Mon, Dec 24, 2012 at 9:34 AM, Kurt Mueller
<kurt.alfred.mueller@gmail.com> wrote:
> $ wget -q -O - http://python.org/ | chardetect.py
> stdin: ISO-8859-2 with confidence 0.803579722043
> $

And it sucks, because it uses magic, and not reading the HTML tags.
The RIGHT thing to do for websites is detect the meta charset
definition, which is

    <meta http-equiv="content-type" content="text/html; charset=utf-8">

or

    <meta charset="utf-8">

The second one for HTML5 websites, and both may require case
conversion and the useless ` /` at the end.  But if somebody is using
HTML5, you are pretty much guaranteed to get UTF-8.

In today’s world, the proper assumption to make is “UTF-8 or GTFO”.
Because nobody in the right mind would use something else today.

-- 
Kwpolska <http://kwpolska.tk>
stop html mail      | always bottom-post
www.asciiribbon.org | www.netmeister.org/news/learn2quote.html
GPG KEY: 5EAAEA16

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

how to detect the character encoding  in a web page ? iMath <redstone-cold@163.com> - 2012-12-23 16:34 -0800
  Re: how to detect the character encoding in a web page ? Chris Angelico <rosuav@gmail.com> - 2012-12-24 12:23 +1100
  Re: how to detect the character encoding  in a web page ? Hans Mulder <hansmu@xs4all.nl> - 2012-12-24 02:30 +0100
  Re: how to detect the character encoding  in a web page ? iMath <redstone-cold@163.com> - 2012-12-23 18:57 -0800
  Re: how to detect the character encoding  in a web page ? iMath <redstone-cold@163.com> - 2012-12-23 19:03 -0800
  Re: how to detect the character encoding  in a web page ? iMath <redstone-cold@163.com> - 2012-12-23 19:03 -0800
    Re: how to detect the character encoding  in a web page ? Kurt Mueller <kurt.alfred.mueller@gmail.com> - 2012-12-24 09:34 +0100
    Re: how to detect the character encoding in a web page ? Kwpolska <kwpolska@gmail.com> - 2012-12-24 13:16 +0100
      Re: how to detect the character encoding in a web page ? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-12-24 13:50 +0000
        Re: how to detect the character encoding in a web page ? Alister <alister.ware@ntlworld.com> - 2012-12-24 16:27 +0000
          Re: how to detect the character encoding in a web page ? Roy Smith <roy@panix.com> - 2012-12-24 11:46 -0500
            Re: how to detect the character encoding in a web page ? albert@spenarnc.xs4all.nl (Albert van der Horst) - 2013-01-14 12:50 +0000
  Re: how to detect the character encoding  in a web page ? python培训 <51mmj.com@gmail.com> - 2012-12-28 06:30 -0800
  Re: how to detect the character encoding  in a web page ? iMath <redstone-cold@163.com> - 2013-01-07 01:23 -0800

csiph-web