Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte

From	Terry Reedy <tjreedy@udel.edu>
Subject	Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte
Date	2013-09-29 13:30 -0400
References	<l28m26$aao$1@dont-email.me> <l28m9v$aao$2@dont-email.me> <mailman.439.1380441619.18130.python-list@python.org> <l28mtv$c50$2@dont-email.me> <524806A7.1050000@nedbatchelder.com>
Newsgroups	comp.lang.python
Message-ID	<mailman.457.1380475864.18130.python-list@python.org> (permalink)

Show all headers | View raw

On 9/29/2013 6:53 AM, Ned Batchelder wrote:

> This is the nature of Unicode pain in Python 2 (Python 3 has a different
> kind!).  This may help you understand what's going on:
> http://nedbatchelder.com/text/unipain.html

This is really excellent and I bookmarked it.

There is one minor error: "the conversion from int to float can't fail,"

 >>> float(10**1000)
Traceback (most recent call last):
   File "<pyshell#0>", line 1, in <module>
     float(10**1000)
OverflowError: long int too large to convert to float

Even when it succeeds, it can fail in the sense of losing information.
 >>> int(float(12345678901234567890))
12345678901234567168
 >>> float(int(1.55))
1.0

This is somewhat analogous to a combination of errors='ignore' and 
errors='replace' (with random garbage).

I think the presentation would be strengthened with the correction, as 
it shows that the problems of conversion are *not* unique to bytes and 
unicode.

-- 
Terry Jan Reedy

Thread

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-09-29 10:49 +0300
  Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-09-29 10:53 +0300
    Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Chris Angelico <rosuav@gmail.com> - 2013-09-29 18:00 +1000
      Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-09-29 11:04 +0300
        Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Chris Angelico <rosuav@gmail.com> - 2013-09-29 18:07 +1000
          Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-09-29 11:19 +0300
            Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Chris Angelico <rosuav@gmail.com> - 2013-09-29 18:23 +1000
              Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-09-29 11:30 +0300
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-09-29 11:37 +0300
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-09-29 09:45 +0000
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-09-29 13:24 +0300
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Dave Angel <davea@davea.name> - 2013-09-29 11:11 +0000
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-09-29 14:25 +0300
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Dave Angel <davea@davea.name> - 2013-09-29 11:46 +0000
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-09-29 14:51 +0300
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Chris “Kwpolska” Warrick <kwpolska@gmail.com> - 2013-09-29 14:38 +0200
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Chris Angelico <rosuav@gmail.com> - 2013-09-30 00:01 +1000
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Chris Angelico <rosuav@gmail.com> - 2013-09-30 00:19 +1000
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-09-29 19:06 +0300
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Joel Goldstick <joel.goldstick@gmail.com> - 2013-09-29 12:14 -0400
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-09-29 19:19 +0300
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Larry Hudson <orgnut@yahoo.com> - 2013-09-29 20:16 -0700
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-09-30 15:45 +0100
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-09-30 21:03 +0300
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2013-09-30 20:18 +0200
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-09-30 19:42 +0100
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-09-30 23:13 +0300
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-09-30 21:44 +0100
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-10-01 00:34 +0300
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Joel Goldstick <joel.goldstick@gmail.com> - 2013-09-30 17:44 -0400
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-10-01 01:08 +0300
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Joel Goldstick <joel.goldstick@gmail.com> - 2013-09-30 18:13 -0400
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-09-30 23:14 +0100
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-10-01 01:19 +0300
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-09-30 23:28 +0100
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-10-01 01:30 +0300
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Chris Angelico <rosuav@gmail.com> - 2013-10-01 08:43 +1000
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-10-01 01:48 +0300
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-09-30 23:50 +0000
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-09-30 22:47 +0100
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Ned Batchelder <ned@nedbatchelder.com> - 2013-09-30 14:54 -0400
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2013-09-30 22:51 +0200
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Chris Angelico <rosuav@gmail.com> - 2013-10-01 08:29 +1000
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-10-01 01:36 +0300
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Chris Angelico <rosuav@gmail.com> - 2013-10-01 08:56 +1000
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-10-01 02:14 +0300
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2013-10-01 09:27 +0200
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-10-01 11:39 +0300
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2013-10-01 11:10 +0200
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-10-02 01:36 +0000
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2013-10-02 09:23 +0200
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-10-02 11:52 +0300
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-10-02 11:49 +0300
                Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Grant Edwards <invalid@invalid.invalid> - 2013-10-01 14:29 +0000
                Mutual respect, tolerance, encouragement (was: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte) Ben Finney <ben+python@benfinney.id.au> - 2013-10-01 08:42 +1000
        Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Ned Batchelder <ned@nedbatchelder.com> - 2013-09-29 06:53 -0400
          Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-09-29 14:08 +0000
        Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Terry Reedy <tjreedy@udel.edu> - 2013-09-29 13:30 -0400
        Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Ned Batchelder <ned@nedbatchelder.com> - 2013-09-29 15:08 -0400
  Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Chris Angelico <rosuav@gmail.com> - 2013-09-29 17:53 +1000
    Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Νίκος <nikos.gr33k@gmail.com> - 2013-09-29 10:59 +0300
      Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-09-29 12:25 -0400
      Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Grant Edwards <invalid@invalid.invalid> - 2013-09-30 13:51 +0000
        Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-09-30 15:51 +0100
          Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Denis McMahon <denismfmcmahon@gmail.com> - 2013-09-30 22:09 +0000
        Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Chris Angelico <rosuav@gmail.com> - 2013-10-01 01:06 +1000

csiph-web