Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path: <ethan@stoneleaf.us>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.000
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'output': 0.05; 'string.': 0.05; 'subject:Python': 0.06; 'string': 0.09; 'from:addr:ethan': 0.09; 'from:addr:stoneleaf.us': 0.09; 'from:name:ethan furman': 0.09; 'message-id:@stoneleaf.us': 0.09; 'received:184.172': 0.09; 'received:gator410.hostgator.com': 0.09; 'subject: [': 0.09; '~ethan~': 0.09; 'python': 0.11; 'cheap.': 0.16; 'correctness.': 0.16; 'iterator': 0.16; 'iterators': 0.16; 'offsets': 0.16; 'subject: \n ': 0.16; 'subject:unicode': 0.16; 'sure.': 0.16; 'language': 0.16; 'wrote:': 0.18; 'not,': 0.20; 'seems': 0.21; 'memory': 0.22; 'header:User-Agent:1': 0.23; 'byte': 0.24; 'mathematical': 0.24; 'prototype': 0.24; '(or': 0.24; "i've": 0.25; 'compare': 0.26; 'subject:/': 0.26; 'header:In-Reply-To:1': 0.27; 'point': 0.28; 'correct': 0.29; "doesn't": 0.30; "i'm": 0.30; 'steven': 0.31; 'languages': 0.32; 'subject:the': 0.34; 'operations': 0.35; 'but': 0.35; 'method': 0.36; 'charset:us- ascii': 0.36; 'starting': 0.37; 'handle': 0.38; 'whatever': 0.38; 'to:addr:python-list': 0.38; 'pm,': 0.38; 'to:addr:python.org': 0.39; 'how': 0.40; 'even': 0.60; 'french': 0.61; 'entire': 0.61; 'received:173': 0.61; 'safe': 0.72; 'walk': 0.74; 'subject:long': 0.84; 'was...': 0.84; 'besides,': 0.93
Date: Thu, 28 Mar 2013 21:56:05 -0700
From: Ethan Furman <ethan@stoneleaf.us>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130106 Thunderbird/17.0.2
MIME-Version: 1.0
To: python-list@python.org
Subject: unicode and the FSR [was: Re: flaming vs accuracy [was Re: Performance of int/long in Python 3]]
References: <mailman.3703.1364248275.2939.python-list@python.org> <a52fbe9d-db14-4ed2-bb49-adfb4b56f973@k4g2000yqn.googlegroups.com> <mailman.3771.1364324590.2939.python-list@python.org> <0b779c80-4f50-4716-8c30-47755c15f304@m12g2000yqp.googlegroups.com> <kit1kg$g2u$1@ger.gmane.org> <nad-98F0A4.17004226032013@news.gmane.org> <kitdqr$4m4$2@ger.gmane.org> <nad-8CB9C0.18315026032013@news.gmane.org> <mailman.3805.1364385073.2939.python-list@python.org> <5153a12d$0$29998$c3e8da3$5496439d@news.astraweb.com> <mailman.3845.1364441182.2939.python-list@python.org> <d2cc443a-e049-42ed-abc6-66b5ea600fe7@j1g2000pbq.googlegroups.com> <mailman.3860.1364451682.2939.python-list@python.org> <987c4bd9-0e5e-4387-9c78-1075a77d3c47@c6g2000yqh.googlegroups.com> <mailman.3863.1364463394.2939.python-list@python.org> <rOednY4OeOjbqcnMnZ2dnUVZ_oWdnZ2d@westnet.com.au> <51543f45$0$29998$c3e8da3$5496439d@news.astraweb.com> <-LGdnWTpyKcdkcjMnZ2dnUVZ_jCdnZ2d@westnet.com.au>
In-Reply-To: <-LGdnWTpyKcdkcjMnZ2dnUVZ_jCdnZ2d@westnet.com.au>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - gator410.hostgator.com
X-AntiAbuse: Original Domain - python.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - stoneleaf.us
X-BWhitelist: no
X-Source: 
X-Source-Args: 
X-Source-Dir: 
X-Source-Sender: ([173.12.184.235]) [173.12.184.235]:56747
X-Source-Auth: ethan+stoneleaf.us
X-Email-Count: 1
X-Source-Cap: dG9idWs7dG9idWs7Z2F0b3I0MTAuaG9zdGdhdG9yLmNvbQ==
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.3936.1364533284.2939.python-list@python.org>
Lines: 30
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1364533284 news.xs4all.nl 6862 [2001:888:2000:d::a6]:45317
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:42222

On 03/28/2013 08:34 PM, Neil Hodgson wrote:
> Steven D'Aprano:
>
>> Any string method that takes a starting offset requires the method to
>> walk the string byte-by-byte. I've even seen languages put responsibility
>> for dealing with that onto the programmer: the "start offset" is given in
>> *bytes*, not characters. I don't remember what language this was... it
>> might have been Haskell? Whatever it was, it horrified me.
>
>     It doesn't horrify me - I've been working this way for over 10 years and it seems completely natural.

Horrifying or not, I am willing to give up a small amount of speed for correctness.  Heck, I'm willing to give up a lot 
of speed for correctness.  Once I have my slow but correct prototype going I can recode in a faster language (if needed) 
and compare it's blazingly fast output with my slowly-generated but known-good output.

>  You can wrap
> access in iterators that hide the byte offsets if you like. This then ensures that all operations on those iterators are
> safe only allowing the iterator to point at the start/end of valid characters.

Sure.  Or I can let Python handle it for me.


>     The counter-problem is that a French document that needs to include one mathematical symbol (or emoji) outside
> Latin-1 will double in size as a Python string.

True.  But how often do you have the entire document as a single string?  Use readlines() instead of read().  Besides, 
memory is cheap.

--
~Ethan~