Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder4.news.weretis.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path: <nedbat@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.004
X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; '(python': 0.07; '*not*': 0.07; 'error:': 0.07; 'float': 0.07; "subject:' ": 0.07; 'subject:position': 0.09; 'cc:addr:python-list': 0.11; 'python': 0.11; 'random': 0.14; 'losing': 0.16; 'reedy': 0.16; 'subject: \n ': 0.16; 'subject:start': 0.16; 'unicode.': 0.16; 'sender:addr:gmail.com': 0.17; 'thanks,': 0.17; 'wrote:': 0.18; '>>>': 0.22; 'cc:addr:python.org': 0.22; 'header:User-Agent:1': 0.23; 'bytes': 0.24; 'unicode': 0.24; 'cc:2**0': 0.24; 'cc:no real name:2**0': 0.24; 'header:In-Reply-To:1': 0.27; 'am,': 0.29; 'nature': 0.30; '"",': 0.31; 'minor': 0.31; 'file': 0.32; '(most': 0.33; '"the': 0.34; 'sense': 0.34; "can't": 0.35; 'convert': 0.35; 'received:google.com': 0.35; 'there': 0.35; 'really': 0.36; 'combination': 0.36; 'shows': 0.36; 'too': 0.37; 'problems': 0.38; 'pm,': 0.38; 'recent': 0.39; 'subject:can': 0.39; 'even': 0.60; 'conversion': 0.61; 'different': 0.65; 'skip:1 20': 0.65; 'batchelder': 0.84; 'correction,': 0.84; 'pain': 0.84; 'strengthened': 0.84
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=C0iS5RsrOQe2MS+DAKSb60XF7sbfABbSs+ah3kXpLAs=; b=gtl/MD56KigknEIJse2b+/U8KQNd9W9WOSH/qSTb3C1DxMNcwgQ7NA1MACp6m2Bxv+ 4Z7sxE8FczoBRX9U0cI5ytfz4wqY8J38M8Ldfuntkzc/cqYRf64zUOkeeTtMEEVwTA+D aeirXSXjSMB+0xZzbUCZDf2d2UIvkXt1aYNqTCDXDvsaMhjAFwU8Vi8j6z5QLSjv2cUW a7fuK+4OLfYzAXkPtmusp9lUcuhknR3pcuRobFKWzTVvWzXM81wc1fcQCWDrXIhAKgh9 bMPsRSMg2FFuZ14wkvmezMbTE/XvV557bmoSfcHGjWse89xIdXXfK/WpzFrcIuiVBfPt X0zA==
X-Received: by 10.224.147.143 with SMTP id l15mr7977qav.113.1380481711025; Sun, 29 Sep 2013 12:08:31 -0700 (PDT)
Sender: Ned Batchelder <nedbat@gmail.com>
Date: Sun, 29 Sep 2013 15:08:28 -0400
From: Ned Batchelder <ned@nedbatchelder.com>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130801 Thunderbird/17.0.8
MIME-Version: 1.0
To: Terry Reedy <tjreedy@udel.edu>
Subject: Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte
References: <l28m26$aao$1@dont-email.me>	<l28m9v$aao$2@dont-email.me> <mailman.439.1380441619.18130.python-list@python.org> <l28mtv$c50$2@dont-email.me> <524806A7.1050000@nedbatchelder.com> <l29o42$m42$1@ger.gmane.org>
In-Reply-To: <l29o42$m42$1@ger.gmane.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: python-list@python.org
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.462.1380481720.18130.python-list@python.org>
Lines: 34
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1380481720 news.xs4all.nl 15987 [2001:888:2000:d::a6]:60472
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:55034

On 9/29/13 1:30 PM, Terry Reedy wrote:
> On 9/29/2013 6:53 AM, Ned Batchelder wrote:
>
>> This is the nature of Unicode pain in Python 2 (Python 3 has a different
>> kind!).  This may help you understand what's going on:
>> http://nedbatchelder.com/text/unipain.html
>
> This is really excellent and I bookmarked it.
>
> There is one minor error: "the conversion from int to float can't fail,"
>
> >>> float(10**1000)
> Traceback (most recent call last):
>   File "<pyshell#0>", line 1, in <module>
>     float(10**1000)
> OverflowError: long int too large to convert to float
>
> Even when it succeeds, it can fail in the sense of losing information.
> >>> int(float(12345678901234567890))
> 12345678901234567168
> >>> float(int(1.55))
> 1.0
>
> This is somewhat analogous to a combination of errors='ignore' and 
> errors='replace' (with random garbage).
>
> I think the presentation would be strengthened with the correction, as 
> it shows that the problems of conversion are *not* unique to bytes and 
> unicode.
>

Thanks, these are excellent points.

--Ned.