Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Sun, 12 Feb 2012 17:40:38 -0500
From: Dave Angel <davea@dejaviewphoto.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.24) Gecko/20111109 Thunderbird/3.1.16
MIME-Version: 1.0
To: Roy Smith <roy@panix.com>
Subject: Re: Python usage numbers
References: <CALFfu7BwqDR+4RN9VNFi35J6HJa2x=rFFTKuGAX_K=J_eRb1xg@mail.gmail.com>	<4F36E2F5.9000505@gmail.com>	<CALFfu7AuYmGURX7=3XgEyHzD+mxtNebF=X-HRjr-LJuodUKpKA@mail.gmail.com>	<mailman.5711.1329010113.27778.python-list@python.org>	<4f37229b$0$29986$c3e8da3$5496439d@news.astraweb.com>	<f610859f-3aa3-4c84-a737-40791a217ef1@m5g2000yqk.googlegroups.com>	<mailman.5715.1329021524.27778.python-list@python.org>	<roy-98B81C.10130712022012@news.panix.com>	<jh9d7i$d6q$1@dough.gmane.org>	<mailman.5739.1329084873.27778.python-list@python.org> <roy-E2270B.17273412022012@news.panix.com>
In-Reply-To: <roy-E2270B.17273412022012@news.panix.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: python-list@python.org
Precedence: list
Reply-To: davea@dejaviewphoto.com
Newsgroups: comp.lang.python
Message-ID: <mailman.5740.1329086473.27778.python-list@python.org>
Lines: 42
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:20302

On 02/12/2012 05:27 PM, Roy Smith wrote:
> In article<mailman.5739.1329084873.27778.python-list@python.org>,
>   Chris Angelico<rosuav@gmail.com>  wrote:
>
>> On Mon, Feb 13, 2012 at 9:07 AM, Terry Reedy<tjreedy@udel.edu>  wrote:
>>> The situation before ascii is like where we ended up *before* unicode.
>>> Unicode aims to replace all those byte encoding and character sets with
>>> *one* byte encoding for *one* character set, which will be a great
>>> simplification. It is the idea of ascii applied on a global rather that
>>> local basis.
>> Unicode doesn't deal with byte encodings; UTF-8 is an encoding, but so
>> are UTF-16, UTF-32. and as many more as you could hope for. But
>> broadly yes, Unicode IS the solution.
> I could hope for one and only one, but I know I'm just going to be
> disapointed.  The last project I worked on used UTF-8 in most places,
> but also used some C and Java libraries which were only available for
> UTF-16.  So it was transcoding hell all over the place.
>
> Hopefully, we will eventually reach the point where storage is so cheap
> that nobody minds how inefficient UTF-32 is and we all just start using
> that.  Life will be a lot simpler then.  No more transcoding, a string
> will just as many bytes as it is characters, and everybody will be happy
> again.

Keep your in-memory character strings as Unicode, and only 
serialize(encode) them when they go to/from a device, or to/from 
anachronistic code.  Then the cost is realized at the point of the 
problem.  No different than when deciding how to serialize any other 
data type.  Do it only at the point of entry/exit of your program.

But as long as devices are addressed as bytes, or as anything smaller 
than 32bit thingies, you will have encoding issues when writing to the 
device, and decoding issues when reading.  At the very least, you have 
big-endian/little-endian ways to encode that UCS-4 code point.