Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.python > #47196
| Path | csiph.com!usenet.pasdenom.info!news.albasani.net!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail |
|---|---|
| Return-Path | <rosuav@gmail.com> |
| X-Original-To | python-list@python.org |
| Delivered-To | python-list@mail.python.org |
| X-Spam-Status | OK 0.005 |
| X-Spam-Evidence | '*H*': 0.99; '*S*': 0.00; 'encoding': 0.05; 'parser': 0.07; 'ascii': 0.09; 'assuming': 0.09; 'bytes.': 0.09; 'http': 0.09; 'indicates': 0.09; 'parsing': 0.09; 'seen,': 0.09; 'sure,': 0.09; 'will,': 0.09; 'assume': 0.14; '8-bit': 0.16; 'content- type': 0.16; 'encodings': 0.16; 'encodings,': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'personally,': 0.16; 'slight': 0.16; 'tag,': 0.16; 'subject: ?': 0.16; 'wrote:': 0.18; 'subject:page': 0.19; 'thu,': 0.19; 'otherwise,': 0.22; 'affects': 0.24; 'bytes': 0.24; 'header': 0.24; "haven't": 0.24; 'header:In-Reply-To:1': 0.27; 'chris': 0.29; 'errors': 0.30; 'message-id:@mail.gmail.com': 0.30; 'this.': 0.32; 'guess': 0.33; 'subject:the': 0.34; "i'd": 0.34; 'problem': 0.35; 'received:209.85': 0.35; 'possible.': 0.35; 'received:209.85.220': 0.35; 'but': 0.35; 'received:google.com': 0.35; 'received:209': 0.37; 'handle': 0.38; 'to:addr:python-list': 0.38; 'pm,': 0.38; 'track': 0.38; 'to:addr:python.org': 0.39; 'tag': 0.61; 'new': 0.61; "you're": 0.61; 'back': 0.62; 'real': 0.63; 'such': 0.63; 'due': 0.66; 'covers': 0.68; 'nobody': 0.68; 'band.': 0.84; 'characters,': 0.84; 'impact.': 0.84; "it'd": 0.84; 'technically': 0.84; 'top.': 0.84; '2013': 0.98 |
| DKIM-Signature | v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=BATSJzrnWkq8Qt3GJAEnboufedk3bWSmbNikB1Cu3i8=; b=XwWFNnjkB1bPrs+4bnX//udLG/JDljlPU3lz0ASbYaWwVZUwg/iKoigSbyzikBtGJ+ ELmFoCcTHNVxmzpmjoEXEuBuYrXygR0CMaVLxoLq7QRxE3ZVnc3/bGvEhF7FboK6N8i5 6zBatPcIyFasSR/vc8oo/nudbknmOW9MuyKZbmwgJxslYGLvtMfMGNDGhlkGSHiABFIy Ei/P96W2x3TbBdS/ZzIsE4DjBcEQh5Bom08Rsy1vEK/AwtJDjx7OKOxSrPqOBOSkmRCw y8mkxieTzx1qF2h/H921euBNYyycmSMyqEYWmiE6Wb+p6M7+5V3ZtnwpcY8NHVYutm46 XMqA== |
| MIME-Version | 1.0 |
| X-Received | by 10.52.65.238 with SMTP id a14mr18246448vdt.24.1370502846460; Thu, 06 Jun 2013 00:14:06 -0700 (PDT) |
| In-Reply-To | <pan.2013.06.06.06.22.36.896000@nowhere.com> |
| References | <c15bad9a-a7f7-456e-8dc5-b1af67fbdd44@googlegroups.com> <29a6b839-1e3d-42ba-acf3-a58a5fcb9f5c@googlegroups.com> <mailman.2752.1370454914.3114.python-list@python.org> <pan.2013.06.06.06.22.36.896000@nowhere.com> |
| Date | Thu, 6 Jun 2013 17:14:06 +1000 |
| Subject | Re: how to detect the character encoding in a web page ? |
| From | Chris Angelico <rosuav@gmail.com> |
| To | python-list@python.org |
| Content-Type | text/plain; charset=ISO-8859-1 |
| X-BeenThere | python-list@python.org |
| X-Mailman-Version | 2.1.15 |
| Precedence | list |
| List-Id | General discussion list for the Python programming language <python-list.python.org> |
| List-Unsubscribe | <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe> |
| List-Archive | <http://mail.python.org/pipermail/python-list/> |
| List-Post | <mailto:python-list@python.org> |
| List-Help | <mailto:python-list-request@python.org?subject=help> |
| List-Subscribe | <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe> |
| Newsgroups | comp.lang.python |
| Message-ID | <mailman.2787.1370502849.3114.python-list@python.org> (permalink) |
| Lines | 31 |
| NNTP-Posting-Host | 2001:888:2000:d::a6 |
| X-Trace | 1370502849 news.xs4all.nl 15962 [2001:888:2000:d::a6]:34586 |
| X-Complaints-To | abuse@xs4all.nl |
| Xref | csiph.com comp.lang.python:47196 |
Show key headers only | View raw
On Thu, Jun 6, 2013 at 4:22 PM, Nobody <nobody@nowhere.com> wrote: > On Thu, 06 Jun 2013 03:55:11 +1000, Chris Angelico wrote: > >> The HTTP header is completely out of band. This is the best way to >> transmit encoding information. Otherwise, you assume 7-bit ASCII and start >> parsing. Once you find a meta tag, you stop parsing and go back to the >> top, decoding in the new way. > > Provided that the meta tag indicates an ASCII-compatible encoding, and you > haven't encountered any decode errors due to 8-bit characters, then > there's no need to go back to the top. Technically and conceptually, you go back to the start and re-parse. Sure, you might optimize that if you can, but not every parser will, hence it's advisable to put the content-type as early as possible. >> "ASCII-compatible" covers a huge number of >> encodings, so it's not actually much of a problem to do this. > > With slight modifications, you can also handle some > almost-ASCII-compatible encodings such as shift-JIS. > > Personally, I'd start by assuming ISO-8859-1, keep track of which bytes > have actually been seen, and only re-start parsing from the top if the > encoding change actually affects the interpretation of any of those bytes. Hrm, it'd be equally valid to guess UTF-8. But as long as you're prepared to re-parse after finding the content-type, that's just a choice of optimization and has no real impact. ChrisA
Back to comp.lang.python | Previous | Next — Previous in thread | Find similar | Unroll thread
Re: how to detect the character encoding in a web page ? iMath <redstone-cold@163.com> - 2013-06-05 08:14 -0700
Re: how to detect the character encoding in a web page ? Chris Angelico <rosuav@gmail.com> - 2013-06-06 03:55 +1000
Re: how to detect the character encoding in a web page ? Nobody <nobody@nowhere.com> - 2013-06-06 07:22 +0100
Re: how to detect the character encoding in a web page ? Chris Angelico <rosuav@gmail.com> - 2013-06-06 17:14 +1000
csiph-web