Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!xlned.com!feeder5.xlned.com!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.053 X-Spam-Evidence: '*H*': 0.89; '*S*': 0.00; 'output': 0.05; 'subject:Python': 0.06; 'dan': 0.09; 'finished.': 0.09; 'main()': 0.09; 'cc:addr:python-list': 0.11; '23,': 0.16; '52,': 0.16; 'cc:name:python list': 0.16; 'codec': 0.16; 'guess.': 0.16; 'guessing': 0.16; 'maintainers': 0.16; 'ought': 0.16; 'skip:g 90': 0.16; 'exception': 0.16; 'so.': 0.16; 'wrote:': 0.18; 'wed,': 0.18; 'result.': 0.19; 'thanks.': 0.20; 'example': 0.22; 'cc:addr:python.org': 0.22; 'byte': 0.24; 'satisfying': 0.24; 'decide': 0.24; '(or': 0.24; 'cc:2**0': 0.24; "i've": 0.25; 'header:In-Reply-To:1': 0.27; 'wondering': 0.29; 'character': 0.29; 'raise': 0.29; 'respective': 0.29; 'subject:) ': 0.29; 'characters': 0.30; 'dec': 0.30; 'skip:& 60': 0.30; 'message- id:@mail.gmail.com': 0.30; "i'm": 0.30; 'getting': 0.31; "d'aprano": 0.31; 'steven': 0.31; 'file': 0.32; 'probably': 0.32; 'text': 0.33; '(most': 0.33; 'guess': 0.33; 'subject: (': 0.35; "can't": 0.35; 'except': 0.35; 'but': 0.35; 'received:google.com': 0.35; 'there': 0.35; 'really': 0.36; 'skip:" 50': 0.36; 'subject:?': 0.36; 'two': 0.37; 'project': 0.37; 'skip:& 10': 0.38; 'pm,': 0.38; 'that,': 0.38; 'recent': 0.39; 'skip:& 20': 0.39; 'skip:. 10': 0.39; '\xa0\xa0\xa0': 0.39; 'skip:u 10': 0.60; 'read': 0.60; 'games,': 0.60; 'skip:c 50': 0.60; 'most': 0.60; 'hope': 0.61; 'full': 0.61; 'entire': 0.61; "you're": 0.61; 'show': 0.63; 'skip:n 10': 0.64; 'our': 0.64; 'movie': 0.65; 'anything.': 0.68; 'ratings': 0.68; 'bulk': 0.74; '^_^': 0.84; 'batchelder': 0.84; 'noise': 0.84; 'rated': 0.84; 'rating.': 0.84; 'subject:Movie': 0.84; '\xa0there': 0.91; 'confidence': 0.95; '2013': 0.98 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=+6A6Qs3iX4McJCLgCdQW6c7UWsF13D4qfnC7nsL6WFo=; b=D0t55CVdqV+FpPEN+xDKg+JQ+qvrZ+h4GSla5PWxMYoDZ2SZBoZALY32icFl8ztMY8 BVTu0kOmB3XVbRTy4/rG98IG9LlFd8/b1AIsCBE2vZsh7W7KS/nqCTwd0S8Y2SjMJJlk RhH0karhwbm3XFqCL8BBoNh7IS7aX4IjWP3zxMTQrVqtP0C+rf2Kln2Picp46FEKikXc XFHYkTipmgEWeH52Ub4XtKSG4XRIEJk+D9ym4BcEDa2sb53GhY1yQqCK3G+KR3fNFrsH HP8zL8lwdClXPt2aHhULJR6RAdCDgATnmoeqkLg7f4cKtuDb1PP8n7vAHuAuOLBXcqJc 8puQ== MIME-Version: 1.0 X-Received: by 10.180.77.49 with SMTP id p17mr26852517wiw.30.1386820340933; Wed, 11 Dec 2013 19:52:20 -0800 (PST) In-Reply-To: References: <2CDEC558-B896-402C-8F70-A3089A5EC93D@gmail.com> <52a8f410$0$29992$c3e8da3$5496439d@news.astraweb.com> Date: Wed, 11 Dec 2013 19:52:20 -0800 Subject: Re: Movie (MPAA) ratings and Python? From: Dan Stromberg To: Ned Batchelder Content-Type: multipart/alternative; boundary=f46d04389213c9eb5304ed4e45e2 Cc: Python List X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 163 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1386820342 news.xs4all.nl 2846 [2001:888:2000:d::a6]:59377 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:61657 --f46d04389213c9eb5304ed4e45e2 Content-Type: text/plain; charset=ISO-8859-1 On Wed, Dec 11, 2013 at 5:01 PM, Ned Batchelder wrote: > On 12/11/13 6:39 PM, Dan Stromberg wrote: > >> >> On Wed, Dec 11, 2013 at 3:24 PM, Steven D'Aprano >> > > wrote: >> >> On Wed, 11 Dec 2013 15:07:35 -0800, Dan Stromberg wrote: >> >> > $ chardet mpaa-ratings-reasons.list >> > mpaa-ratings-reasons.list: windows-1255 (confidence: 0.97) >> > >> > I'm aware that chardet is playing guessing games, though one >> would hope >> > it would guess well most of the time, and give a reasonable >> confidence >> > rating. >> >> What reason do you have for thinking that Windows-1255 isn't a >> reasonable >> guess? If the bulk of the text is Latin-1 except perhaps for one or >> two >> Hebrew characters (or what chardet thinks are Hebrew characters), it >> may >> actually be a reasonable guess. >> >> >> I get a traceback if I try to read the file as Windows-1255. I don't >> get a traceback if I read it as ISO-8859-1. >> >> If it is a poor guess, perhaps you ought to report it to the chardet >> maintainers as a good example of a poor guess. >> >> I was considering that, and may do so. >> >> I've also been wondering if ISO-8859-1 is just an octet-oriented codec, >> so it'll read about anything. There are clearly non-7-bit-ASCII >> characters in the file that look like line noise in an mrxvt. >> > > Both ISO-8859-1 and Windows-1255 are octet-oriented, I don't see why one > would raise an exception when the other didn't. Unless the exception isn't > on the decode, but instead on your attempt to output the result. Can you > show the full traceback you're seeing? > $ ./movie-ratings Traceback (most recent call last): File "./movie-ratings", line 85, in main() File "./movie-ratings", line 68, in main ratings = get_ratings('/home/dstromberg/src/home-svn/movie-ratings/trunk/mpaa-ratings-reasons.list') File "./movie-ratings", line 52, in get_ratings for line in ratings_file: File "/usr/local/cpython-3.3/lib/python3.3/encodings/cp1255.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0xfc in position 1225: character maps to BTW, other than satisfying our respective curiosities, I consider this project finished. It's probably not getting ratings for my entire movie collection, but it is getting them for a significant fraction, which is all I was really looking for. Now I know which ones are rated PG, so I can decide whether to let my 8 year old watch them. This is with cpython-3.3. Thanks. ^_^ --f46d04389213c9eb5304ed4e45e2 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

= On Wed, Dec 11, 2013 at 5:01 PM, Ned Batchelder <ned@nedbatchelder.com= > wrote:
On 12/1= 1/13 6:39 PM, Dan Stromberg wrote:

On Wed, Dec 11, 2013 at 3:24 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info
<mailto:steve+comp.lang.python@pearwood.info>> wrote:<= br>
=A0 =A0 On Wed, 11 Dec 2013 15:07:35 -0800, Dan Stromberg wrote:

=A0 =A0 =A0> =A0$ chardet mpaa-ratings-reasons.list
=A0 =A0 =A0> mpaa-ratings-reasons.list: windows-1255 (confidence: 0.97)<= br> =A0 =A0 =A0>
=A0 =A0 =A0> I'm aware that chardet is playing guessing games, thoug= h one
=A0 =A0 would hope
=A0 =A0 =A0> it would guess well most of the time, and give a reasonable=
=A0 =A0 confidence
=A0 =A0 =A0> rating.

=A0 =A0 What reason do you have for thinking that Windows-1255 isn't a<= br> =A0 =A0 reasonable
=A0 =A0 guess? If the bulk of the text is Latin-1 except perhaps for one or= two
=A0 =A0 Hebrew characters (or what chardet thinks are Hebrew characters), i= t may
=A0 =A0 actually be a reasonable guess.


I get a traceback if I try to read the file as Windows-1255. =A0I don't=
get a traceback if I read it as ISO-8859-1.

=A0 =A0 If it is a poor guess, perhaps you ought to report it to the charde= t
=A0 =A0 maintainers as a good example of a poor guess.

I was considering that, and may do so.

I've also been wondering if ISO-8859-1 is just an octet-oriented codec,=
so it'll read about anything. =A0There are clearly non-7-bit-ASCII
characters in the file that look like line noise in an mrxvt.

Both ISO-8859-1 and Windows-1255 are octet-oriented, I don't see why on= e would raise an exception when the other didn't. =A0Unless the excepti= on isn't on the decode, but instead on your attempt to output the resul= t. Can you show the full traceback you're seeing?

$ ./mov= ie-ratings
Traceback (most recent call last):
=A0 File "./movie-= ratings", line 85, in <module>
=A0=A0=A0 main()
=A0 File &= quot;./movie-ratings", line 68, in main
=A0=A0=A0 ratings =3D get_ratings('/home/dstromberg/src/home-svn/movie-= ratings/trunk/mpaa-ratings-reasons.list')
=A0 File "./movie-rat= ings", line 52, in get_ratings
=A0=A0=A0 for line in ratings_file:<= br>=A0 File "/usr/local/cpython-3.3/lib/python3.3/encodings/cp1255.py&= quot;, line 23, in decode
=A0=A0=A0 return codecs.charmap_decode(input,self.errors,decoding_table)[0]=
UnicodeDecodeError: 'charmap' codec can't decode byte 0xfc = in position 1225: character maps to <undefined>

BTW= , other than satisfying our respective curiosities, I consider this project= finished.=A0 It's probably not getting ratings for my entire movie col= lection, but it is getting them for a significant fraction, which is all I = was really looking for.=A0 Now I know which ones are rated PG, so I can dec= ide whether to let my 8 year old watch them.

This is with cpython-3.3.

Thanks.=A0 ^_^
--f46d04389213c9eb5304ed4e45e2--