Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!selfless.tophat.at!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.005 X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'slices': 0.07; 'python': 0.08; 'counting': 0.09; 'received:mail-bw0-f46.google.com': 0.09; 'string;': 0.09; '>>>': 0.12; 'account)': 0.16; 'modes': 0.16; 'narrow': 0.16; 'received:209.85.214.46': 0.16; 'skip:r 50': 0.16; 'subject:Unicode': 0.16; 'surrogate': 0.16; 'surrogates': 0.16; '\xc2\xa0i': 0.16; 'header:In-Reply-To:1': 0.21; 'code': 0.24; 'index': 0.25; '(and': 0.25; 'url:mailman': 0.26; 'string': 0.26; 'thanks.': 0.27; 'message-id:@mail.gmail.com': 0.28; 'received:209.85.214': 0.28; 'import': 0.29; 'unicode': 0.29; 'instead': 0.29; "won't": 0.30; 'url:listinfo': 0.30; 'characters,': 0.30; 'item,': 0.30; 'separated': 0.30; 'hi,': 0.31; 'print': 0.31; 'agree': 0.32; 'to:addr:python-list': 0.33; 'list': 0.33; 'points': 0.34; 'characters': 0.34; 'there': 0.35; 'probably': 0.36; 'received:google.com': 0.37; 'received:209.85': 0.37; 'two': 0.37; 'url:python': 0.38; 'url:org': 0.38; 'subject:: ': 0.38; 'received:209': 0.39; 'to:addr:python.org': 0.39; 'your': 0.60; 'wide': 0.61; 'marks,': 0.84; 'points,': 0.91; 'realy': 0.91 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type:content-transfer-encoding; bh=i06hFCMHVdY0sl2Y9BycyrpFgdsM9eVmlgzAW+QUIX8=; b=xG2Ntru3Dcsnx9ltInBSFAmEtNf44Vc2MFFxYaNbzv9Utv2fd9U5naygQRbD2pFtm+ hdR666f/tddSdz+SpCYCLV99E5hzwejZsmf9naBughaYhJ3DmLpG2blWqH2iSYtWDBF7 fTvqDOG0tZCpG/firvxwZserZvWVSLgOTkXiA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=lKoKdHujkcnFHRG6hTw9ijvcum/4VGjcV5qnqgDoIYIhpKwHD6OE0LWTU1Cyp2gEey VxZA32RD6Yf6mbNFO+pBxVMBjoIYLjOVLEkf6cR3UndQ/iJdXIIRVDT8FmSRZjLbFyVh 6kPK0gKW9G9UrEU8EfAMlAndwtoAx44t5Ci9Q= MIME-Version: 1.0 In-Reply-To: References: Date: Thu, 23 Jun 2011 00:15:19 +0200 Subject: Re: Unicode codepoints From: Vlastimil Brom To: python-list@python.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 39 NNTP-Posting-Host: 82.94.164.166 X-Trace: 1308780921 news.xs4all.nl 14138 [::ffff:82.94.164.166]:39917 X-Complaints-To: abuse@xs4all.nl Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:8246 2011/6/22 Saul Spatz : > Thanks. =C2=A0I agree with you about the generator. =C2=A0Using your firs= t suggestion, code points above U+FFFF get separated into two "surrogate pa= ir" characters fron UTF-16. =C2=A0So instead of U=3D10FFFF I get U+DBFF and= U+DFFF. > -- > http://mail.python.org/mailman/listinfo/python-list > Hi, If you realy need the wide unicode functionality on a narrow unicode python build and only need to get the string index of characters including surrogate pairs counting as one item, you can build a list of single characters or surrogate pairs, e.g.: >>> surrog_txt=3Du"a=F0=90=8C=B0 =F0=90=8C=B1 =F0=90=8C=B2 =F0=90=8C=B3" >>> surrog_txt u'a\U00010330 \U00010331 \U00010332 \U00010333' >>> print surrog_txt a=F0=90=8C=B0 =F0=90=8C=B1 =F0=90=8C=B2 =F0=90=8C=B3 >>> list(surrog_txt) [u'a', u'\ud800', u'\udf30', u' ', u'\ud800', u'\udf31', u' ', u'\ud800', u'\udf32', u' ', u'\ud800', u'\udf33'] >>> import re >>> re.findall(ur"(?s)(?:[\ud800-\udbff][\udc00-\udfff])|.", surrog_txt) [u'a', u'\U00010330', u' ', u'\U00010331', u' ', u'\U00010332', u' ', u'\U00010333'] >>> this way, the indices, slices and len() would work on the supplementary list as expected for a normal string; however it probably won't be very efficient for longer texts. Note that surrogates are not the only asymmetry between code points, characters (and glyphs - to take the visual representation of those into account) - there are combining diacritical marks, in various combinations with precomposed diacritical characters, multiple normalisation modes etc. regards, vbr