Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!news.mixmin.net!hq-usenetpeers.eweka.nl!hq-usenetpeers.eweka.nl!bcyclone01.am1.xlned.com!bcyclone01.am1.xlned.com!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <546e880d$0$12995$c3e8da3$5496439d@news.astraweb.com>
References: <m4kcu8$ta8$1@ger.gmane.org> <m4kn66$ma3$1@ger.gmane.org> <1416491999.209592.193345213.3A6FF9A4@webmail.messagingengine.com> <m4l20h$toc$1@ger.gmane.org> <CAPTjJmqjUGmTA-0JXrkeKYLeShmT=mOE=c38o8YD7XheCK_5Kw@mail.gmail.com> <m4l53q$ff4$1@ger.gmane.org> <mailman.16029.1416502283.18130.python-list@python.org> <87ppch8zvx.fsf@elektro.pacujo.net> <546e880d$0$12995$c3e8da3$5496439d@news.astraweb.com>
Date: Fri, 21 Nov 2014 12:10:12 +1100
Subject: Re: python 2.7 and unicode (one more time)
From: Chris Angelico <rosuav@gmail.com>
Cc: "python-list@python.org" <python-list@python.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.16072.1416532215.18130.python-list@python.org>
Lines: 129
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:81156

On Fri, Nov 21, 2014 at 11:32 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> (E.g. there are millions of existing files across the world containing te=
xt
> which use legacy encodings that are not compatible with Unicode.)

Not compatible with Unicode? There aren't many character sets out
there that include characters not in Unicode - that was the whole
point. Of course, there are plenty of files in unspecified eight-bit
encodings, so you may have a problem with reliable decoding - but if
you know what the encoding is, you ought to be able to represent each
character in Unicode.

Not compatible with any of the UTFs, that's different. Plenty of that
in the world.

> You are certainly correct that in it's full generality, "text" is much mo=
re
> than just a string of code points. Unicode strings is a primitive data
> type. A powerful and sophisticated text processing application may even
> find Python strings too primitive, possibly needing something like ropes =
of
> graphemes rather than strings of code points.

That's probably more an efficiency point, though. It should be
possible to do a perfect two-way translation between your grapheme
rope and a Python string; otherwise, you'll have great difficulty
saving your file to the disk (which will normally involve representing
the text in Unicode, then encoding that to bytes).

To be sure, a Python string is a poor representational form for a text
editor. But that's largely because it's immutable, so every little
edit would involve massive copying. Depending on what you're doing, it
might be worth using a chunked UTF-8 byte stream (allowing for
insertion at any chunk boundary), or an array of lines, or something
grapheme-based... but all of those questions are performance, not
correctness, issues.

> We Western and Northern European speakers -- and I don't know whether Fin=
ns
> are counted as Northern Europeans or Eastern Europeans -- are lucky in th=
at
> our natural languages are well-covered by Unicode. All our graphemes are
> also code points, even the "funny ones with accents". As an English
> speaker. I have to remind myself that not every grapheme is a single code
> point, but Devanagari or Navajo writers will never make that mistake.

I've been working with different languages a bit, lately. Broadly
speaking, you have:

1) Languages which use the Roman alphabet, plus a handful of other
characters (eg Finnish, German). These can be represented largely in
ASCII, and used to be handled fairly easily with a single codepage -
an eight-bit ASCII-compatible encoding.

2) Languages which use a different alphabet (eg Cyrillic - Russian,
Bulgarian). You could possibly cram them into an eight-bit encoding
without tipping ASCII out, but I'm not sure. In Unicode, these
languages are all easily supported by the BMP, as they don't use a
huge number of characters each.

3) Languages which use a non-alphabetic system (eg Korean). I think
they're all still covered by the BMP, but there's no way you can fit
them into eight-bit encodings - one single language will use more than
256 symbols.

4) Ancient, esoteric, or symbolic writing systems. Not fundamentally
different from the above categories except that they're less used, and
the BMP has finite space. These will definitely need the SMP.

But all of them are covered by Unicode. (Sadly, they are NOT all
covered by all fonts, so I've been finding that certain pieces of text
come out as strings of little boxes. But I can at least manipulate the
text, even if I can't read it back.) I can, for example, zip lines of
text like this:

English:
Let it go, let it go!
I am one with the wind and sky
Let it go, let it go!
You'll never see me cry!

Icelandic:
=C3=9Eetta er n=C3=B3g, =C3=BEetta er n=C3=B3g
Uppi =C3=AD himni eins og vindabl=C3=A6r
=C3=9Eetta er n=C3=B3g, komi=C3=B0 n=C3=B3g
Og t=C3=A1r m=C3=ADn enginn s=C3=A9r f=C3=A6r

Russian:
=D0=9E=D1=82=D0=BF=D1=83=D1=81=D1=82=D0=B8 =D0=B8 =D0=B7=D0=B0=D0=B1=D1=83=
=D0=B4=D1=8C,
=D0=AD=D1=82=D0=BE=D1=82 =D0=BC=D0=B8=D1=80 =D0=B8=D0=B7 =D1=82=D0=B2=D0=BE=
=D0=B8=D1=85 =D0=B3=D1=80=D1=91=D0=B7.
=D0=9E=D1=82=D0=BF=D1=83=D1=81=D1=82=D0=B8 =D0=B8 =D0=B7=D0=B0=D0=B1=D1=83=
=D0=B4=D1=8C,
=D0=98 =D0=BD=D0=B5 =D0=B1=D1=83=D0=B4=D0=B5=D1=82 =D0=B1=D0=BE=D0=BB=D1=8C=
=D1=88=D0=B5 =D1=81=D0=BB=D1=91=D0=B7.


Output:
Let it go, let it go!
=C3=9Eetta er n=C3=B3g, =C3=BEetta er n=C3=B3g
=D0=9E=D1=82=D0=BF=D1=83=D1=81=D1=82=D0=B8 =D0=B8 =D0=B7=D0=B0=D0=B1=D1=83=
=D0=B4=D1=8C,

I am one with the wind and sky
Uppi =C3=AD himni eins og vindabl=C3=A6r
=D0=AD=D1=82=D0=BE=D1=82 =D0=BC=D0=B8=D1=80 =D0=B8=D0=B7 =D1=82=D0=B2=D0=BE=
=D0=B8=D1=85 =D0=B3=D1=80=D1=91=D0=B7.

Let it go, let it go!
=C3=9Eetta er n=C3=B3g, komi=C3=B0 n=C3=B3g
=D0=9E=D1=82=D0=BF=D1=83=D1=81=D1=82=D0=B8 =D0=B8 =D0=B7=D0=B0=D0=B1=D1=83=
=D0=B4=D1=8C,

You'll never see me cry!
Og t=C3=A1r m=C3=ADn enginn s=C3=A9r f=C3=A6r
=D0=98 =D0=BD=D0=B5 =D0=B1=D1=83=D0=B4=D0=B5=D1=82 =D0=B1=D0=BE=D0=BB=D1=8C=
=D1=88=D0=B5 =D1=81=D0=BB=D1=91=D0=B7.


In fact, it's trivially easy to write something like this, because all
this text is Unicode. ALL of these languages (and plenty more) are
"well-covered by Unicode". There's still the ongoing debate of Han
unification, plus the progressive work of adding characters for
ancient scripts and such, but AFAIK, all writing systems currently in
use are covered.

ChrisA