Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #89870

Re: Unicode surrogate pairs (Python 3.4)

Path csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!1.eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path <rosuav@gmail.com>
X-Original-To python-list@python.org
Delivered-To python-list@mail.python.org
X-Spam-Status OK 0.013
X-Spam-Evidence '*H*': 0.97; '*S*': 0.00; 'encoding': 0.05; 'subject:Python': 0.06; 'string': 0.09; 'bytes,': 0.09; 'cc:addr :python-list': 0.11; 'python': 0.11; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'ideally,': 0.16; 'ought': 0.16; 'pairs': 0.16; 'subject:Unicode': 0.16; 'surrogate': 0.16; 'fix': 0.17; 'wrote:': 0.18; 'result.': 0.19; 'not,': 0.20; 'appears': 0.22; 'cc:addr:python.org': 0.22; 'form:': 0.24; "shouldn't": 0.24; 'unicode': 0.24; 'mon,': 0.24; 'cc:2**0': 0.24; 'skip:" 20': 0.27; 'header:In-Reply-To:1': 0.27; 'am,': 0.29; 'characters': 0.30; 'message-id:@mail.gmail.com': 0.30; 'there.': 0.32; 'subject: (': 0.35; 'convert': 0.35; 'no,': 0.35; 'but': 0.35; 'received:google.com': 0.35; 'possible': 0.36; 'how': 0.40; 'even': 0.60; 'catch': 0.60; 'conversion': 0.61; "you're": 0.61; 'first': 0.61; 'kind': 0.63; 'skip:n 10': 0.64; 'between': 0.67; 'combining': 0.68; 'containing': 0.69; '2015': 0.84; 'absolutely': 0.87; 'to:none': 0.92
DKIM-Signature v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-type; bh=vHbcIsoP8C/gTfJR3nRp5F16ddzyJkOcU9ju2tnBdL8=; b=CBVK2G0lifK3uO3vVGxPifEpQMU9GLZ6PrQnuLb1KiDaZNhV73ZQhO83tCoRcEltFz NkxXoY3RwlwSwhWafGrk6UI0ViTTqpwgSzZvppbBdVbdsUV1Hzws1o9nOKWgnLF9XsXY hEcnN/j5099CGtWXKqCpdDavKOtINanv5Vhy+usELEA2llwxhEEA8Gl5ppe7LL1zoEDc x6hjPWTrNoSdmvnIlnklFMXDBTFqq62TPIJvi/YCuUcMvFzihlxazr5DluOtj/wFtdIf LXqGCQy0VUaHcthlXdG9ICbMAG1ovpWxBNlzmQ8xHdHzv7VWZOxI59F3KXa1c3+EVgzJ 2KeQ==
MIME-Version 1.0
X-Received by 10.50.114.35 with SMTP id jd3mr8308307igb.14.1430665532014; Sun, 03 May 2015 08:05:32 -0700 (PDT)
In-Reply-To <slrnmkccs4.apd.jon+usenet@frosty.unequivocal.co.uk>
References <slrnmkccs4.apd.jon+usenet@frosty.unequivocal.co.uk>
Date Mon, 4 May 2015 01:05:31 +1000
Subject Re: Unicode surrogate pairs (Python 3.4)
From Chris Angelico <rosuav@gmail.com>
Cc "python-list@python.org" <python-list@python.org>
Content-Type text/plain; charset=UTF-8
X-BeenThere python-list@python.org
X-Mailman-Version 2.1.20+
Precedence list
List-Id General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive <http://mail.python.org/pipermail/python-list/>
List-Post <mailto:python-list@python.org>
List-Help <mailto:python-list-request@python.org?subject=help>
List-Subscribe <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups comp.lang.python
Message-ID <mailman.67.1430665534.12865.python-list@python.org> (permalink)
Lines 24
NNTP-Posting-Host 2001:888:2000:d::a6
X-Trace 1430665534 news.xs4all.nl 2936 [2001:888:2000:d::a6]:40508
X-Complaints-To abuse@xs4all.nl
Xref csiph.com comp.lang.python:89870

Show key headers only | View raw

On Mon, May 4, 2015 at 12:40 AM, Jon Ribbens
<jon+usenet@unequivocal.co.uk> wrote:
> If I have a string containing surrogate pairs like this in Python 3.4:
>
>   "\udb40\udd9d"
>
> How do I convert it into the proper form:
>
>   "\U000E019D"
>
> ? The answer appears not to be "unicodedata.normalize".

No, it's not, because Unicode normalization is a very specific thing.
You're looking for a fix for some kind of encoding issue; Unicode
normalization translates between combining characters and combined
characters.

You shouldn't even actually _have_ those in your string in the first
place. How did you construct/receive that data? Ideally, catch it at
that point, and deal with it there. But if you absolutely have to
convert the surrogates, it ought to be possible to do a sloppy UCS-2
conversion to bytes, then a proper UTF-16 decode on the result.

ChrisA

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Unicode surrogate pairs (Python 3.4) Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2015-05-03 14:40 +0000
  Re: Unicode surrogate pairs (Python 3.4) Chris Angelico <rosuav@gmail.com> - 2015-05-04 01:05 +1000
    Re: Unicode surrogate pairs (Python 3.4) Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2015-05-03 15:32 +0000
      Re: Unicode surrogate pairs (Python 3.4) Marko Rauhamaa <marko@pacujo.net> - 2015-05-03 18:35 +0300
      Re: Unicode surrogate pairs (Python 3.4) Chris Angelico <rosuav@gmail.com> - 2015-05-04 01:48 +1000
        Re: Unicode surrogate pairs (Python 3.4) Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2015-05-03 16:30 +0000
          Re: Unicode surrogate pairs (Python 3.4) Chris Angelico <rosuav@gmail.com> - 2015-05-04 02:47 +1000
      Re: Unicode surrogate pairs (Python 3.4) MRAB <python@mrabarnett.plus.com> - 2015-05-03 16:53 +0100
        Re: Unicode surrogate pairs (Python 3.4) Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2015-05-03 16:26 +0000
          Re: Unicode surrogate pairs (Python 3.4) MRAB <python@mrabarnett.plus.com> - 2015-05-03 18:09 +0100
            Re: Unicode surrogate pairs (Python 3.4) Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2015-05-03 19:20 +0000

csiph-web