Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!ecngs!feeder2.ecngs.de!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.001 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'bytes.': 0.07; 'escape': 0.07; 'users,': 0.07; 'python': 0.09; 'any.': 0.09; 'byte,': 0.09; 'creator': 0.09; 'lawrence': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'subject:string': 0.09; 'stored': 0.10; 'language': 0.14; '"pay"': 0.16; '3.3,': 0.16; '4-byte': 0.16; 'benefit.': 0.16; 'design:': 0.16; 'grounds': 0.16; 'hint:': 0.16; 'received:80.91.229.3': 0.16; 'received:plane.gmane.org': 0.16; 'saying.': 0.16; 'subject:unicode': 0.16; 'string': 0.17; 'wrote:': 0.17; 'bytes': 0.17; 'detect': 0.17; 'tend': 0.17; 'unicode': 0.17; 'saying': 0.18; 'memory': 0.18; 'preferred': 0.20; 'required.': 0.22; 'least': 0.25; 'header:In-Reply-To:1': 0.25; 'header:User-Agent:1': 0.26; 'correct': 0.28; 'header:X -Complaints-To:1': 0.28; 'rest': 0.28; 'overhead': 0.29; 'represented': 0.29; 'sleep': 0.29; 'character': 0.29; 'points': 0.29; "i'm": 0.29; "skip:' 10": 0.30; 'code': 0.31; 'point': 0.31; 'could': 0.32; 'getting': 0.33; 'hopefully': 0.33; 'to:addr :python-list': 0.33; 'me?': 0.33; 'wrong': 0.34; 'posting': 0.35; 'table': 0.35; 'there': 0.35; 'received:org': 0.36; 'characters': 0.36; 'test': 0.36; 'correctly': 0.37; 'does': 0.37; 'received:co.za': 0.37; 'received:za': 0.37; 'subject:: ': 0.38; 'mark': 0.38; 'performance': 0.39; 'to:addr:python.org': 0.39; 'google': 0.39; 'where': 0.40; 'header:Received:5': 0.40; 'think': 0.40; 'subject:, ': 0.61; 'world': 0.63; 'subject:...': 0.63; 'email addr:gmail.com': 0.63; 'making': 0.64; 'saturday': 0.65; 'soon': 0.70; 'benefit': 0.70; 'saving': 0.72; 'received:41': 0.73; 'frank': 0.75; '"das': 0.84; 'grosse': 0.84; 'overhead,': 0.84; 'points,': 0.84; 'subject:, ...': 0.84; 'water.': 0.84 X-Injected-Via-Gmane: http://gmane.org/ To: python-list@python.org From: Frank Millman Subject: Re: Flexible string representation, unicode, typography, ... Date: Sat, 25 Aug 2012 11:46:34 +0200 References: <1874857c-68ef-4c1b-b15a-46ef47df9445@googlegroups.com> <1cb3f062-eb45-4b0c-977b-76afb099923c@googlegroups.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Gmane-NNTP-Posting-Host: 41-133-114-138.dsl.mweb.co.za User-Agent: Mozilla/5.0 (Windows NT 5.2; rv:14.0) Gecko/20120713 Thunderbird/14.0 In-Reply-To: X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 46 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1345888006 news.xs4all.nl 6960 [2001:888:2000:d::a6]:46747 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:27860 On 25/08/2012 10:58, Mark Lawrence wrote: > On 25/08/2012 08:27, wxjmfauth@gmail.com wrote: >> >> Unicode design: a flat table of code points, where all code >> points are "equals". >> As soon as one attempts to escape from this rule, one has to >> "pay" for it. >> The creator of this machinery (flexible string representation) >> can not even benefit from it in his native language (I think >> I'm correctly informed). >> >> Hint: Google -> "Das grosse Eszett" >> >> jmf >> > > It's Saturday morning, I'm stone cold sober, had a good sleep and I'm > still baffled as to the point if any. Could someone please enlightem me? > Here's what I think he is saying. I am posting this to test the water. I am also confused, and if I have got it wrong hopefully someone will correct me. In python 3.3, unicode strings are now stored as follows - if all characters can be represented by 1 byte, the entire string is composed of 1-byte characters else if all characters can be represented by 1 or 2 bytea, the entire string is composed of 2-byte characters else the entire string is composed of 4-byte characters There is an overhead in making this choice, to detect the lowest number of bytes required. jmfauth believes that this only benefits 'english-speaking' users, as the rest of the world will tend to have strings where at least one character requires 2 or 4 bytes. So they incur the overhead, without getting any benefit. Therefore, I think he is saying that he would have preferred that python standardise on 4-byte characters, on the grounds that the saving in memory does not justify the performance overhead. Frank Millman