Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #81214

Re: python 2.7 and unicode (one more time)

Path csiph.com!usenet.pasdenom.info!goblin2!goblin.stu.neva.ru!newsfeed.xs4all.nl!newsfeed2a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path <rosuav@gmail.com>
X-Original-To python-list@python.org
Delivered-To python-list@mail.python.org
X-Spam-Status OK 0.000
X-Spam-Evidence '*H*': 1.00; '*S*': 0.00; 'languages.': 0.04; 'encoding': 0.05; 'insert': 0.05; '21,': 0.07; 'detect': 0.07; 'encoded': 0.07; 'plenty': 0.07; '22,': 0.09; 'ascii': 0.09; 'assuming': 0.09; 'cursor': 0.09; 'subject:2.7': 0.09; 'cc:addr :python-list': 0.11; 'python': 0.11; '"code': 0.16; '11:32': 0.16; 'boundaries,': 0.16; 'codec': 0.16; 'encodings': 0.16; 'encodings,': 0.16; 'enough.': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'ordinals': 0.16; 'ought': 0.16; 'programmers.': 0.16; 'stem': 0.16; 'subject:unicode': 0.16; 'trivially': 0.16; 'unicode,': 0.16; 'unicode.': 0.16; 'unicode?': 0.16; 'words.': 0.16; 'subject:python': 0.16; 'files.': 0.16; 'sat,': 0.16; 'wrote:': 0.18; 'bit': 0.19; 'widget': 0.19; 'meant': 0.20; '>>>': 0.22; 'issue.': 0.22; 'saying': 0.22; 'cc:addr:python.org': 0.22; 'this?': 0.23; "aren't": 0.24; 'library,': 0.24; 'string,': 0.24; 'text,': 0.24; 'unicode': 0.24; 'visible': 0.24; 'earlier': 0.24; 'cc:2**0': 0.24; 'equivalent': 0.26; 'least': 0.26; 'header:In-Reply-To:1': 0.27; 'chris': 0.29; 'am,': 0.29; 'character': 0.29; 'related': 0.29; 'characters': 0.30; 'sets': 0.30; 'message-id:@mail.gmail.com': 0.30; 'code': 0.31; "skip:' 10": 0.31; 'concern': 0.31; 'concern.': 0.31; "d'aprano": 0.31; 'steven': 0.31; 'though.': 0.31; 'yes.': 0.31; 'file': 0.32; 'compatible': 0.32; 'quite': 0.32; 'text': 0.33; '(e.g.': 0.33; 'fri,': 0.33; 'subject:time': 0.33; 'sense': 0.34; "i'd": 0.34; 'problem': 0.35; 'subject: (': 0.35; "can't": 0.35; 'something': 0.35; 'convert': 0.35; 'editor': 0.35; 'point.': 0.35; 'but': 0.35; 'received:google.com': 0.35; 'there': 0.35; 'data,': 0.36; 'disk': 0.36; 'subject:one': 0.36; "didn't": 0.36; 'easily': 0.37; 'represent': 0.38; 'problems': 0.38; 'nov': 0.38; 'files': 0.38; 'ability': 0.39; 'expect': 0.39; 'even': 0.60; 'read': 0.60; 'middle': 0.60; 'problems.': 0.60; 'most': 0.60; 'year.': 0.61; "you're": 0.61; 'subject:more': 0.64; 'more': 0.64; 'world': 0.66; 'positions': 0.67; 'between': 0.67; 'combining': 0.68; 'containing': 0.69; 'obvious': 0.74; 'dealt': 0.91; 'to:none': 0.92
DKIM-Signature v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-type; bh=0OFMPUURouOaFbCwPWvh9abQhWF+vTFlsYXzrPDOUBQ=; b=rCZdZVXJbWTa/re4JNMV9phPri1uKLY5P/5nDzkD+3oY41Y4TVSaZ/IzuYmmv0beyM HVtdjceNd+j5khLsQEGCIxZpch9PSpAAOWRLwTyj+d/iViRr4gCXQN5Ih1VOHGlaKvcR VQyXVDLdzXOFUe4+xdRjNDf4jZZJ83ylfXXveZ8q6cDddd5ZwyTUferrwOYoAJoGjKtI Gx+KSNbANlPaUJUmOwpYA4dUTjdd87ReqE8JVm+7nvCTboTYmNVEkhMGi1oQxknjIEQr 7fXaRLSKCEX99iusAP7Cer4ak8K2CzQIYJMolDAn8k4dmyCTs1s1w2l33MTpyCEzJlMV rGOw==
MIME-Version 1.0
X-Received by 10.42.110.195 with SMTP id r3mr13488554icp.12.1416584443188; Fri, 21 Nov 2014 07:40:43 -0800 (PST)
In-Reply-To <546f58db$0$12990$c3e8da3$5496439d@news.astraweb.com>
References <m4kcu8$ta8$1@ger.gmane.org> <m4kn66$ma3$1@ger.gmane.org> <1416491999.209592.193345213.3A6FF9A4@webmail.messagingengine.com> <m4l20h$toc$1@ger.gmane.org> <CAPTjJmqjUGmTA-0JXrkeKYLeShmT=mOE=c38o8YD7XheCK_5Kw@mail.gmail.com> <m4l53q$ff4$1@ger.gmane.org> <mailman.16029.1416502283.18130.python-list@python.org> <87ppch8zvx.fsf@elektro.pacujo.net> <546e880d$0$12995$c3e8da3$5496439d@news.astraweb.com> <mailman.16072.1416532215.18130.python-list@python.org> <546f58db$0$12990$c3e8da3$5496439d@news.astraweb.com>
Date Sat, 22 Nov 2014 02:40:43 +1100
Subject Re: python 2.7 and unicode (one more time)
From Chris Angelico <rosuav@gmail.com>
Cc "python-list@python.org" <python-list@python.org>
Content-Type text/plain; charset=UTF-8
X-BeenThere python-list@python.org
X-Mailman-Version 2.1.15
Precedence list
List-Id General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive <http://mail.python.org/pipermail/python-list/>
List-Post <mailto:python-list@python.org>
List-Help <mailto:python-list-request@python.org?subject=help>
List-Subscribe <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups comp.lang.python
Message-ID <mailman.16105.1416584445.18130.python-list@python.org> (permalink)
Lines 53
NNTP-Posting-Host 2001:888:2000:d::a6
X-Trace 1416584445 news.xs4all.nl 2877 [2001:888:2000:d::a6]:42350
X-Complaints-To abuse@xs4all.nl
Xref csiph.com comp.lang.python:81214

Show key headers only | View raw

On Sat, Nov 22, 2014 at 2:23 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Chris Angelico wrote:
>
>> On Fri, Nov 21, 2014 at 11:32 AM, Steven D'Aprano
>> <steve+comp.lang.python@pearwood.info> wrote:
>>> (E.g. there are millions of existing files across the world containing
>>> text which use legacy encodings that are not compatible with Unicode.)
>>
>> Not compatible with Unicode? There aren't many character sets out
>> there that include characters not in Unicode - that was the whole
>> point. Of course, there are plenty of files in unspecified eight-bit
>> encodings, so you may have a problem with reliable decoding - but if
>> you know what the encoding is, you ought to be able to represent each
>> character in Unicode.
>
> What I meant was that some encodings -- namely ASCII and Latin-1 -- the
> ordinals are exactly equivalent to Unicode, that is:
>
> That's not quite as significant as I thought, though. What is significant is
> that a pure ASCII file on disk can be read by a program assuming UTF-8:
>
> although the same is not the case for Latin-1 encoded files.

Yep. Thing is, Unicode can't magically convert all files on all
disks... but with a good codec library, you can at least convert
things as you find them. (I was reading MacRoman files earlier this
year. THAT is an encoding I didn't expect I'd find in 2014.)

> Well, yes. My point, agreeing with Marko, is that any time you want to do
> something even vaguely related to human-readable text, "code points" are
> not enough. ... What about something like this?
>
> 'w\N{COMBINING CIRCUMFLEX ACCENT}\N{COMBINING OGONEK}\N{COMBINING CARON}'
>
> If I insert a character into my string, I want to be able to insert before
> the w or after the caron, but not in the middle of those three code points.

Yes, which is a concern. Also a concern is the ability to detect other
boundaries, like words. None of these can be easily solved; all of
them can be dealt with by using the Unicode character data, which is
better than you get for most legacy encodings. In terms of Python
strings, it still makes sense to insert characters between those
combining characters; so what you're saying is that a text editor
widget needs to be aware of more than just code points. Which is
trivially obvious in the presence of RTL text, too; cursor positions
through differing-direction text will be an issue.

The problems you're citing aren't Unicode problems. They stem from the
complexities of human languages. Unicode just makes them a bit more
visible to English-only-speaking programmers.

ChrisA

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Re: python 2.7 and unicode (one more time) Michael Torrie <torriem@gmail.com> - 2014-11-20 09:51 -0700
  Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-20 20:56 +0200
    Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-21 10:49 +1100
      Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-21 08:36 +0200
        Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-21 18:02 +1100
          Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-21 09:14 +0200
            Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-21 18:29 +1100
              Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-21 10:16 +0200
                Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-21 19:24 +1100
        Re: python 2.7 and unicode (one more time) Rustom Mody <rustompmody@gmail.com> - 2014-11-21 08:03 -0800
          Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-21 18:36 +0200
            Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-22 03:52 +1100
            Re: python 2.7 and unicode (one more time) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-22 15:38 +1100
              Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-22 08:40 +0200
                Re: python 2.7 and unicode (one more time) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-23 00:50 +1100
                Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-23 01:17 +1100
                Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-22 16:18 +0200
                Re: python 2.7 and unicode (one more time) Roy Smith <roy@panix.com> - 2014-11-22 09:44 -0500
                Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-22 16:56 +0200
                Re: python 2.7 and unicode (one more time) wxjmfauth@gmail.com - 2014-11-22 07:44 -0800
                Re: python 2.7 and unicode (one more time) Rustom Mody <rustompmody@gmail.com> - 2014-11-22 07:57 -0800
                Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-22 19:49 +0200
                Re: python 2.7 and unicode (one more time) Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-11-22 18:17 +0000
                Re: python 2.7 and unicode (one more time) wxjmfauth@gmail.com - 2014-11-22 11:32 -0800
                Re: python 2.7 and unicode (one more time) wxjmfauth@gmail.com - 2014-11-22 11:17 -0800
                Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-23 07:17 +1100
                Re: python 2.7 and unicode (one more time) Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-11-22 22:04 +0000
                Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-23 09:31 +1100
                Re: python 2.7 and unicode (one more time) Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-11-22 23:38 +0000
                Re: python 2.7 and unicode (one more time) wxjmfauth@gmail.com - 2014-11-23 04:52 -0800
                Re: python 2.7 and unicode (one more time) random832@fastmail.us - 2014-11-22 20:52 -0500
                Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-23 13:11 +1100
                Re: python 2.7 and unicode (one more time) random832@fastmail.us - 2014-11-22 22:10 -0500
                Re: python 2.7 and unicode (one more time) Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-11-23 11:33 -0500
                Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-24 03:39 +1100
                Re: python 2.7 and unicode (one more time) random832@fastmail.us - 2014-11-23 13:13 -0500
                Re: python 2.7 and unicode (one more time) Dave Angel <d@davea.name> - 2014-11-23 15:31 -0500
                Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-24 09:22 +1100
                Re: python 2.7 and unicode (one more time) Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-11-23 20:24 -0500
                Re: python 2.7 and unicode (one more time) random832@fastmail.us - 2014-11-24 00:05 -0500
                Re: python 2.7 and unicode (one more time) Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2014-11-24 11:51 +1300
                Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-24 09:59 +1100
                Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-24 08:57 +0200
                Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-24 18:18 +1100
                Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-24 09:40 +0200
                Re: python 2.7 and unicode (one more time) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-25 09:56 +1100
                Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-25 10:08 +1100
                Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-25 08:23 +0200
                Re: python 2.7 and unicode (one more time) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-25 22:56 +1100
                Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-26 00:07 +1100
                Re: python 2.7 and unicode (one more time) wxjmfauth@gmail.com - 2014-11-25 06:26 -0800
              Re: python 2.7 and unicode (one more time) random832@fastmail.us - 2014-11-22 20:35 -0500
                Re: python 2.7 and unicode (one more time) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-23 17:17 +1100
                Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-23 17:37 +1100
                Re: python 2.7 and unicode (one more time) wxjmfauth@gmail.com - 2014-11-23 07:40 -0800
    Re: python 2.7 and unicode (one more time) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-21 11:32 +1100
      Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-21 12:10 +1100
        Re: python 2.7 and unicode (one more time) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-22 02:23 +1100
          Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-22 02:40 +1100
          Re: python 2.7 and unicode (one more time) Tim Chase <python.list@tim.thechases.com> - 2014-11-21 10:00 -0600
      Re: python 2.7 and unicode (one more time) random832@fastmail.us - 2014-11-20 20:31 -0500
      Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-21 12:39 +1100

csiph-web