Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed1a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Robin Becker <robin@reportlab.com>
Subject: Re: Micro Python -- a lean and efficient implementation of Python 3
Date: Wed, 04 Jun 2014 12:53:19 +0100
References: <CANw+MznPsKgJiW6e_O370VUsmVVxBfQ=M_7WUyU7+wNh+-qefA@mail.gmail.com> <CAPTjJmoB0eMMMhjUz++yYV2CEv=2xUXx7P8UuRvCk7y7gB-4+Q@mail.gmail.com> <20140603194949.3147497d@x34f> <CAPTjJmrwGVaJKmzLiX8buZQmGxrGJV657Jnb7fsK7j1-pLxtVA@mail.gmail.com> <mailman.10646.1401831682.18130.python-list@python.org> <44acd692-5dcd-4e5f-8238-7fbe0de4db2a@googlegroups.com> <mailman.10673.1401853976.18130.python-list@python.org> <c04434ce-cbc4-49ab-b312-24f1631dd894@googlegroups.com> <mailman.10684.1401866176.18130.python-list@python.org> <538ecdef$0$11109$c3e8da3@news.astraweb.com> <7xoay9w1h0.fsf@ruckus.brouhaha.com> <20140604060152.2cdf8198@bigbox.christie.dr>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:24.0) Gecko/20100101 Thunderbird/24.5.0
In-Reply-To: <20140604060152.2cdf8198@bigbox.christie.dr>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.10699.1401882811.18130.python-list@python.org>
Lines: 45
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:72628

On 04/06/2014 12:01, Tim Chase wrote:
> On 2014-06-04 00:58, Paul Rubin wrote:
>> Steven D'Aprano <steve@pearwood.info> writes:
>>>> Maybe there's a use-case for a microcontroller that works in
>>>> ISO-8859-5 natively, thus using only eight bits per character,
>>> That won't even make the Russians happy, since in Russia there
>>> are multiple incompatible legacy encodings.
>>
>> I've never understood why not use UTF-8 for everything.
>
> If you use UTF-8 for everything, then you end up in a world where
> string-indexing (see ChrisA's other side thread on this topic) is no
> longer an O(1) operation, but an O(N) operation.  Some of us slice
> strings for a living. ;-)  I understand that using UTF-32 would allow
> us to maintain O(1) indexing at the cost of every string occupying 4
> bytes per character.  The FSR (again, as I understand it) allows
> strings that fit in one-byte-per-character to use that, scaling up to
> use wider characters internally as they're actually needed/used.
>
........
I believe that we should distinguish between glyph/character indexing and string 
indexing. Even in unicode it may be hard to decide where a visual glyph starts 
and ends. I assume most people would like to assign one glyph to one unicode, 
but that's not always possible with composed glyphs.

 >>> for a in (u'\xc5',u'A\u030a'):
... 	for o in (u'\xf6',u'o\u0308'):
... 		u=a+u'ngstr'+o+u'm'
... 		print("%s %s" % (repr(u),u))
...
u'\xc5ngstr\xf6m' Ångström
u'\xc5ngstro\u0308m' Ångström
u'A\u030angstr\xf6m' Ångström
u'A\u030angstro\u0308m' Ångström
 >>> u'\xc5ngstr\xf6m'==u'\xc5ngstro\u0308m'
False

so even unicode doesn't always allow for O(1) glyph indexing. I know this is 
artificial, but this is the same situation as utf8 faces just the frequency of 
occurrence is different. A very large amount of computing is still western 
centric so searching a byte string for latin characters is still efficient; 
searching for an n with a tilde on top might not be so easy.
-- 
Robin Becker