Path: csiph.com!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Terry Reedy <tjreedy@udel.edu>
Subject: Re: Encoding of surrogate code points to UTF-8
Date: Tue, 08 Oct 2013 17:47:15 -0400
References: <52540e03$0$29984$c3e8da3$5496439d@news.astraweb.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.0
In-Reply-To: <52540e03$0$29984$c3e8da3$5496439d@news.astraweb.com>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.876.1381268848.18130.python-list@python.org>
Lines: 84
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:56446

On 10/8/2013 9:52 AM, Steven D'Aprano wrote:
> I think this is a bug in Python's UTF-8 handling, but I'm not sure.
>
> If I've read the Unicode FAQs correctly, you cannot encode *lone*
> surrogate code points into UTF-8:
>
> http://www.unicode.org/faq/utf_bom.html#utf8-5
>
> Sure enough, using Python 3.3:
>
> py> surr = '\udc80'

I am pretty sure that if Python were being strict, that would raise an 
error, as the result is not a valid unicode string. Allowing the above 
or not was debated and laxness was allowed for at least the following 
practical reasons.

1. Python itself uses the invalid surrogate codepoints for 
surrogateescape error-handling.
http://www.python.org/dev/peps/pep-0383/

2. Invalid strings are needed for tests ;-)
-- like the one you do next.

3. Invalid strings may be needed for interfacing with other C APIs.

> py> surr.encode('utf-8')
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> UnicodeEncodeError: 'utf-8' codec can't encode character '\udc80' in
> position 0: surrogates not allowed

Default strict encoding (utf-8 or otherwise) will only encode valid 
unicode strings. Encode invalid strings with surrogate codepoints with 
surrogateescape error handling.

> But reading the previous entry in the FAQs:
>
> http://www.unicode.org/faq/utf_bom.html#utf8-4
>
> I interpret this as meaning that I should be able to encode valid pairs
> of surrogates.

It says you should be able to 'convert' them, and that the result for 
utf-8 encoding must be a single 4-bytes code for the corresponding 
supplementary codepoint.

> So if I find a code point that encodes to a surrogate pair
> in UTF-16:
>
> py> c = '\N{LINEAR B SYLLABLE B038 E}'
> py> surr_pair = c.encode('utf-16be')
> py> print(surr_pair)
> b'\xd8\x00\xdc\x01'
>
> and then use those same values as the code points, I ought to be able to
> encode to UTF-8, as if it were the same \N{LINEAR B SYLLABLE B038 E} code
> point. But I can't:
>
> py> s = '\ud800\udc01'

This is now a string with two invalid codepoints instead of one ;-).
As above, it would be rejected if Python were being strict.

> py> s.encode('utf-8')
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in
> position 0: surrogates not allowed
>
>
> Have I misunderstood? I think that Python is being too strict about
> rejecting surrogate code points.

No, it is being too lax about allowing them at all.

I believe there is an issue on the tracker (maybe closed) about the doc 
for unicode escapes in string literals. Perhaps is should say more 
clearly that inserting surrogates is allowed but results in an invalid 
string that cannot be normally encoded.

-- 
Terry Jan Reedy