Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed2a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Fri, 21 Nov 2014 10:00:10 -0600
From: Tim Chase <python.list@tim.thechases.com>
To: python-list@python.org
Subject: Re: python 2.7 and unicode (one more time)
In-Reply-To: <546f58db$0$12990$c3e8da3$5496439d@news.astraweb.com>
References: <m4kcu8$ta8$1@ger.gmane.org> <m4kn66$ma3$1@ger.gmane.org> <1416491999.209592.193345213.3A6FF9A4@webmail.messagingengine.com> <m4l20h$toc$1@ger.gmane.org> <CAPTjJmqjUGmTA-0JXrkeKYLeShmT=mOE=c38o8YD7XheCK_5Kw@mail.gmail.com> <m4l53q$ff4$1@ger.gmane.org> <mailman.16029.1416502283.18130.python-list@python.org> <87ppch8zvx.fsf@elektro.pacujo.net> <546e880d$0$12995$c3e8da3$5496439d@news.astraweb.com> <mailman.16072.1416532215.18130.python-list@python.org> <546f58db$0$12990$c3e8da3$5496439d@news.astraweb.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.16107.1416585466.18130.python-list@python.org>
Lines: 60
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:81218

On 2014-11-22 02:23, Steven D'Aprano wrote:
> LATIN SMALL LETTER E
> COMBINING CIRCUMFLEX ACCENT
>=20
> then my application should treat that as a single "character" and
> display it as:
>=20
> LATIN SMALL LETTER E WITH CIRCUMFLEX
>=20
> which looks like this: =C3=AA
>=20
> rather than two distinct "characters" e=CB=86
>=20
> Now, that specific example is a no-brainer, because the Unicode
> normalization routines will handle the conversion. But not every
> combination of accented characters has a canonical combined form.
> What about something like this?
>=20
> 'w\N{COMBINING CIRCUMFLEX ACCENT}\N{COMBINING OGONEK}\N{COMBINING
> CARON}'
>=20
> If I insert a character into my string, I want to be able to insert
> before the w or after the caron, but not in the middle of those
> three code points.

Things get even weirder if you have

 '\N{LATIN SMALL LETTER E WITH CIRCUMFLEX}\N{COMBINING
 OGONEK}\N{COMBINING CARON}'

and when you try to do comparisons like

 s1 =3D '\N{LATIN SMALL LETTER E WITH CIRCUMFLEX}\N{COMBINING OGONEK}'
 s2 =3D 'e\N{COMBINING CIRCUMFLEX ACCENT}\N{COMBINING OGONEK}'
 s3 =3D 'e\N{COMBINING OGONEK}\N{COMBINING CIRCUMFLEX ACCENT}'
 print(s1 =3D=3D s2)
 print(s1 =3D=3D s3)
 print(s2 =3D=3D s3)

Then you also have the case where you want to edit text and the user
wants to remove the COMBINING OGONEK from the character, so you *do*
want to do something akin to

 s4 =3D ''.join(c for c in s3 if c !=3D '\N{COMBINING OGONEK}')

And yet, weird things happen if you try to remove the circumflex:

  for test in (s1, s2, s3):
    print(test =3D=3D ''.join(
      c for c in test if c !=3D '\N{COMBINING CIRCUMFLEX ACCENT}'
      )

They all make sense if you understand what's going on under the hood,
but from a visual/conceptual perspective, something feels amiss.

-tkc