Re: Unicode codepoints

Path	csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!selfless.tophat.at!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path	<vlastimil.brom@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.005
X-Spam-Evidence	'H': 0.99; 'S': 0.00; 'slices': 0.07; 'python': 0.08; 'counting': 0.09; 'received:mail-bw0-f46.google.com': 0.09; 'string;': 0.09; '>>>': 0.12; 'account)': 0.16; 'modes': 0.16; 'narrow': 0.16; 'received:209.85.214.46': 0.16; 'skip:r 50': 0.16; 'subject:Unicode': 0.16; 'surrogate': 0.16; 'surrogates': 0.16; '\xc2\xa0i': 0.16; 'header:In-Reply-To:1': 0.21; 'code': 0.24; 'index': 0.25; '(and': 0.25; 'url:mailman': 0.26; 'string': 0.26; 'thanks.': 0.27; 'message-id:@mail.gmail.com': 0.28; 'received:209.85.214': 0.28; 'import': 0.29; 'unicode': 0.29; 'instead': 0.29; "won't": 0.30; 'url:listinfo': 0.30; 'characters,': 0.30; 'item,': 0.30; 'separated': 0.30; 'hi,': 0.31; 'print': 0.31; 'agree': 0.32; 'to:addr:python-list': 0.33; 'list': 0.33; 'points': 0.34; 'characters': 0.34; 'there': 0.35; 'probably': 0.36; 'received:google.com': 0.37; 'received:209.85': 0.37; 'two': 0.37; 'url:python': 0.38; 'url:org': 0.38; 'subject:: ': 0.38; 'received:209': 0.39; 'to:addr:python.org': 0.39; 'your': 0.60; 'wide': 0.61; 'marks,': 0.84; 'points,': 0.91; 'realy': 0.91
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type:content-transfer-encoding; bh=i06hFCMHVdY0sl2Y9BycyrpFgdsM9eVmlgzAW+QUIX8=; b=xG2Ntru3Dcsnx9ltInBSFAmEtNf44Vc2MFFxYaNbzv9Utv2fd9U5naygQRbD2pFtm+ hdR666f/tddSdz+SpCYCLV99E5hzwejZsmf9naBughaYhJ3DmLpG2blWqH2iSYtWDBF7 fTvqDOG0tZCpG/firvxwZserZvWVSLgOTkXiA=
DomainKey-Signature	a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=lKoKdHujkcnFHRG6hTw9ijvcum/4VGjcV5qnqgDoIYIhpKwHD6OE0LWTU1Cyp2gEey VxZA32RD6Yf6mbNFO+pBxVMBjoIYLjOVLEkf6cR3UndQ/iJdXIIRVDT8FmSRZjLbFyVh 6kPK0gKW9G9UrEU8EfAMlAndwtoAx44t5Ci9Q=
MIME-Version	1.0
In-Reply-To	<bec262ea-9690-4efa-8ae0-1657b46af58d@glegroupsg2000goo.googlegroups.com>
References	<mailman.267.1308715226.1164.python-list@python.org> <bec262ea-9690-4efa-8ae0-1657b46af58d@glegroupsg2000goo.googlegroups.com>
Date	Thu, 23 Jun 2011 00:15:19 +0200
Subject	Re: Unicode codepoints
From	Vlastimil Brom <vlastimil.brom@gmail.com>
To	python-list@python.org
Content-Type	text/plain; charset=UTF-8
Content-Transfer-Encoding	quoted-printable
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.12
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.302.1308780921.1164.python-list@python.org> (permalink)
Lines	39
NNTP-Posting-Host	82.94.164.166
X-Trace	1308780921 news.xs4all.nl 14138 [::ffff:82.94.164.166]:39917
X-Complaints-To	abuse@xs4all.nl
Xref	x330-a1.tempe.blueboxinc.net comp.lang.python:8246

Show key headers only | View raw

2011/6/22 Saul Spatz <saul.spatz@gmail.com>:
> Thanks.  I agree with you about the generator.  Using your first suggestion, code points above U+FFFF get separated into two "surrogate pair" characters fron UTF-16.  So instead of U=10FFFF I get U+DBFF and U+DFFF.
> --
> http://mail.python.org/mailman/listinfo/python-list
>
Hi,
If you realy need the wide unicode functionality on a narrow unicode
python build and only need to get the string index of characters
including surrogate pairs counting as one item, you can build a list
of single characters or surrogate pairs, e.g.:

>>> surrog_txt=u"a𐌰 𐌱 𐌲 𐌳"
>>> surrog_txt
u'a\U00010330 \U00010331 \U00010332 \U00010333'
>>> print surrog_txt
a𐌰 𐌱 𐌲 𐌳
>>> list(surrog_txt)
[u'a', u'\ud800', u'\udf30', u' ', u'\ud800', u'\udf31', u' ',
u'\ud800', u'\udf32', u' ', u'\ud800', u'\udf33']
>>> import re
>>> re.findall(ur"(?s)(?:[\ud800-\udbff][\udc00-\udfff])|.", surrog_txt)
[u'a', u'\U00010330', u' ', u'\U00010331', u' ', u'\U00010332', u' ',
u'\U00010333']
>>>

this way, the indices, slices and len() would work on the
supplementary list as expected for a normal string; however it
probably won't be very efficient for longer texts.
Note that surrogates are not the only asymmetry between code points,
characters (and glyphs - to take the visual representation of those
into account) - there are combining diacritical marks, in various
combinations with precomposed diacritical characters, multiple
normalisation modes etc.

regards,
   vbr

Back to comp.lang.python | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Re: Unicode codepoints Saul Spatz <saul.spatz@gmail.com> - 2011-06-22 06:43 -0700
  Re: Unicode codepoints Vlastimil Brom <vlastimil.brom@gmail.com> - 2011-06-23 00:15 +0200

csiph-web