Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!1.eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path: <python@mrabarnett.plus.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.003
X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'encoding': 0.05; 'subject:Python': 0.06; 'encoded': 0.07; 'string': 0.09; 'bytes,': 0.09; 'happen.': 0.09; 'url:unicode': 0.09; 'python': 0.11; 'from:addr:mrabarnett.plus.com': 0.16; 'from:addr:python': 0.16; 'from:name:mrab': 0.16; 'ideally,': 0.16; 'message- id:@mrabarnett.plus.com': 0.16; 'ought': 0.16; 'pairs': 0.16; 'received:192.168.1.4': 0.16; 'subject:Unicode': 0.16; 'surrogate': 0.16; 'unlikely': 0.16; 'fix': 0.17; 'wrote:': 0.18; 'trying': 0.19; 'result.': 0.19; 'not,': 0.20; 'seems': 0.21; '>>>': 0.22; 'appears': 0.22; 'saying': 0.22; 'header:User- Agent:1': 0.23; 'form:': 0.24; "shouldn't": 0.24; 'unicode': 0.24; 'mon,': 0.24; 'looks': 0.24; 'skip:" 20': 0.27; 'header:In-Reply- To:1': 0.27; 'appear': 0.29; 'chris': 0.29; 'am,': 0.29; "doesn't": 0.30; 'characters': 0.30; "i'm": 0.30; 'there.': 0.32; 'guess': 0.33; 'subject: (': 0.35; 'convert': 0.35; 'no,': 0.35; 'but': 0.35; 'in:': 0.36; 'possible': 0.36; 'url:org': 0.36; 'to:addr:python-list': 0.38; 'little': 0.38; 'support,': 0.39; 'to:addr:python.org': 0.39; 'how': 0.40; 'even': 0.60; 'skip:u 10': 0.60; 'read': 0.60; 'catch': 0.60; 'conversion': 0.61; 'url:3': 0.61; "you're": 0.61; 'first': 0.61; 'kind': 0.63; 'skip:n 10': 0.64; 'url:0': 0.67; 'between': 0.67; 'combining': 0.68; 'containing': 0.69; '2015': 0.84; 'absolutely': 0.87
X-CM-Score: 0.00
X-CNFS-Analysis: v=2.1 cv=DZWZq5dW c=1 sm=1 tr=0 a=0nF1XD0wxitMEM03M9B4ZQ==:117 a=0nF1XD0wxitMEM03M9B4ZQ==:17 a=0Bzu9jTXAAAA:8 a=SY9aFQKHIdAA:10 a=IkcTkHD0fZMA:10 a=EBOSESyhAAAA:8 a=pGLkceISAAAA:8 a=Lpnxw2taAAAA:8 a=te1EGT4yAAAA:8 a=p-Iw2RACd4HDK46Ah9oA:9 a=QEXdDO2ut3YA:10
X-AUTH: mrabarnett@:2500
Date: Sun, 03 May 2015 16:53:47 +0100
From: MRAB <python@mrabarnett.plus.com>
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: Unicode surrogate pairs (Python 3.4)
References: <slrnmkccs4.apd.jon+usenet@frosty.unequivocal.co.uk> <mailman.67.1430665534.12865.python-list@python.org> <slrnmkcftt.230.jon+usenet@frosty.unequivocal.co.uk>
In-Reply-To: <slrnmkcftt.230.jon+usenet@frosty.unequivocal.co.uk>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.20+
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.69.1430668429.12865.python-list@python.org>
Lines: 38
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1430668429 news.xs4all.nl 2929 [2001:888:2000:d::a6]:36426
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:89877

On 2015-05-03 16:32, Jon Ribbens wrote:
> On 2015-05-03, Chris Angelico <rosuav@gmail.com> wrote:
>> On Mon, May 4, 2015 at 12:40 AM, Jon Ribbens
>><jon+usenet@unequivocal.co.uk> wrote:
>>> If I have a string containing surrogate pairs like this in Python 3.4:
>>>
>>>   "\udb40\udd9d"
>>>
>>> How do I convert it into the proper form:
>>>
>>>   "\U000E019D"
>>>
>>> ? The answer appears not to be "unicodedata.normalize".
>>
>> No, it's not, because Unicode normalization is a very specific thing.
>> You're looking for a fix for some kind of encoding issue; Unicode
>> normalization translates between combining characters and combined
>> characters.
>>
>> You shouldn't even actually _have_ those in your string in the first
>> place. How did you construct/receive that data? Ideally, catch it at
>> that point, and deal with it there.
>
> That would, unfortunately, be "tell the Unicode Consortium to format
> their documents differently", which seems unlikely to happen. I'm
> trying to read in: http://www.unicode.org/Public/idna/6.3.0/IdnaTest.txt
>
That document looks like it's encoded in UTF-8.

>> But if you absolutely have to convert the surrogates, it ought to be
>> possible to do a sloppy UCS-2 conversion to bytes, then a proper
>> UTF-16 decode on the result.
>
> Python doesn't appear to have UCS-2 support, so I guess what you're
> saying is that I have to write my own surrogate-decoder? This seems
> a little surprising.
>