Path: csiph.com!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder2.enfer-du-nord.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Terry Reedy <tjreedy@udel.edu>
Subject: Re: Why are some unicode error handlers "encode only"?
Date: Sun, 11 Mar 2012 13:10:33 -0400
References: <4f5cb8c2$0$29891$c3e8da3$5496439d@news.astraweb.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0
In-Reply-To: <4f5cb8c2$0$29891$c3e8da3$5496439d@news.astraweb.com>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.570.1331485852.3037.python-list@python.org>
Lines: 60
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:21499

On 3/11/2012 10:37 AM, Steven D'Aprano wrote:
> At least two standard error handlers are documented as working for
> encoding only:
>
> xmlcharrefreplace
> backslashreplace
>
> See http://docs.python.org/library/codecs.html#codec-base-classes
>
> and http://docs.python.org/py3k/library/codecs.html
>
> Why is this?

I presume the purpose of both is to facilitate transmission of unicode=20
text via byte transmission by extending incomplete byte encodings by=20
replacing unicode chars that do not fit in the given encoding by a ascii =

byte sequence that will fit.

> I don't see why they shouldn't work for decoding as well.
> Consider this example using Python 3.2:
>
>>>> b"aaa--\xe9z--\xe9!--bbb".decode("cp932")
> Traceback (most recent call last):
>    File "<stdin>", line 1, in<module>
> UnicodeDecodeError: 'cp932' codec can't decode bytes in position 9-10:
> illegal multibyte sequence
>
> The two bytes b'\xe9!' is an illegal multibyte sequence for CP-932 (als=
o
> known as MS-KANJI or SHIFT-JIS). Is there some reason why this shouldn'=
t
> or can't be supported?
>
> # This doesn't actually work.
> b"aaa--\xe9z--\xe9!--bbb".decode("cp932", "backslashreplace")
> =3D>  r'aaa--=E9=A8=B7--\xe9\x21--bbb'

This output does not round-trip and would be a bit of a fib since it=20
somewhat misrepresents what the encoded bytes were:

 >>> r'aaa--=E9=A8=B7--\xe9\x21--bbb'.encode("cp932")
b'aaa--\xe9z--\\xe9\\x21--bbb'
 >>> b'aaa--\xe9z--\\xe9\\x21--bbb'.decode("cp932")
'aaa--=E9=A8=B7--\\xe9\\x21--bbb'

Python 3 added surrogateescape error handling to solve this problem.

> and similarly for xmlcharrefreplace.

Since xml character references are representations of unicode chars, and =

not bytes, I do not see how that would work. By analogy, perhaps you=20
mean to have '&#e9;&#21;' in your output instead of '\xe9\x21', but=20
those would not properly be xml numeric character references.

--=20
Terry Jan Reedy