Path: csiph.com!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.001 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'bytes.': 0.07; 'escape': 0.07; 'users,': 0.07; 'python': 0.09; 'any.': 0.09; 'byte,': 0.09; 'creator': 0.09; 'lawrence': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'subject:string': 0.09; 'terry': 0.09; 'stored': 0.10; 'language': 0.14; 'cases': 0.15; '"pay"': 0.16; '3.3,': 0.16; '4-byte': 0.16; 'benefit.': 0.16; 'design:': 0.16; 'grounds': 0.16; 'hint:': 0.16; 'received:80.91.229.3': 0.16; 'received:plane.gmane.org': 0.16; 'reedy': 0.16; 'saying.': 0.16; 'subject:unicode': 0.16; 'string': 0.17; 'wrote:': 0.17; 'bytes': 0.17; 'detect': 0.17; 'tend': 0.17; 'unicode': 0.17; 'saying': 0.18; '>>>': 0.18; 'memory': 0.18; 'preferred': 0.20; 'required.': 0.22; 'least': 0.25; 'header:In-Reply-To:1': 0.25; 'header:User- Agent:1': 0.26; 'correct': 0.28; 'header:X-Complaints-To:1': 0.28; 'rest': 0.28; 'overhead': 0.29; 'represented': 0.29; 'sleep': 0.29; 'character': 0.29; 'points': 0.29; "i'm": 0.29; "skip:' 10": 0.30; 'code': 0.31; 'point': 0.31; 'from:addr:yahoo.co.uk': 0.32; 'could': 0.32; 'getting': 0.33; 'hopefully': 0.33; 'substantial': 0.33; 'to:addr:python-list': 0.33; 'me?': 0.33; 'wrong': 0.34; 'posting': 0.35; 'table': 0.35; 'there': 0.35; 'received:org': 0.36; 'characters': 0.36; 'test': 0.36; 'correctly': 0.37; 'does': 0.37; 'subject:: ': 0.38; 'mark': 0.38; 'performance': 0.39; 'to:addr:python.org': 0.39; 'google': 0.39; 'where': 0.40; 'header:Received:5': 0.40; 'think': 0.40; 'subject:, ': 0.61; 'world': 0.63; 'subject:...': 0.63; 'email addr:gmail.com': 0.63; 'making': 0.64; 'therefore': 0.65; 'saturday': 0.65; 'soon': 0.70; 'benefit': 0.70; 'saving': 0.72; 'frank': 0.75; 'savings': 0.75; '"das': 0.84; 'grosse': 0.84; 'overhead,': 0.84; 'points,': 0.84; 'subject:, ...': 0.84; 'water.': 0.84; 'worthwhile.': 0.84; 'received:89': 0.86 X-Injected-Via-Gmane: http://gmane.org/ To: python-list@python.org From: Mark Lawrence Subject: Re: Flexible string representation, unicode, typography, ... Date: Sat, 25 Aug 2012 12:05:08 +0100 References: <1874857c-68ef-4c1b-b15a-46ef47df9445@googlegroups.com> <1cb3f062-eb45-4b0c-977b-76afb099923c@googlegroups.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Gmane-NNTP-Posting-Host: host-89-240-160-192.as13285.net User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:14.0) Gecko/20120713 Thunderbird/14.0 In-Reply-To: X-Antivirus: avast! (VPS 120824-2, 24/08/2012), Outbound message X-Antivirus-Status: Clean X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 57 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1345892703 news.xs4all.nl 6971 [2001:888:2000:d::a6]:40082 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:27865 On 25/08/2012 10:46, Frank Millman wrote: > On 25/08/2012 10:58, Mark Lawrence wrote: >> On 25/08/2012 08:27, wxjmfauth@gmail.com wrote: >>> >>> Unicode design: a flat table of code points, where all code >>> points are "equals". >>> As soon as one attempts to escape from this rule, one has to >>> "pay" for it. >>> The creator of this machinery (flexible string representation) >>> can not even benefit from it in his native language (I think >>> I'm correctly informed). >>> >>> Hint: Google -> "Das grosse Eszett" >>> >>> jmf >>> >> >> It's Saturday morning, I'm stone cold sober, had a good sleep and I'm >> still baffled as to the point if any. Could someone please enlightem me? >> > > Here's what I think he is saying. I am posting this to test the water. I > am also confused, and if I have got it wrong hopefully someone will > correct me. > > In python 3.3, unicode strings are now stored as follows - > if all characters can be represented by 1 byte, the entire string is > composed of 1-byte characters > else if all characters can be represented by 1 or 2 bytea, the entire > string is composed of 2-byte characters > else the entire string is composed of 4-byte characters > > There is an overhead in making this choice, to detect the lowest number > of bytes required. > > jmfauth believes that this only benefits 'english-speaking' users, as > the rest of the world will tend to have strings where at least one > character requires 2 or 4 bytes. So they incur the overhead, without > getting any benefit. > > Therefore, I think he is saying that he would have preferred that python > standardise on 4-byte characters, on the grounds that the saving in > memory does not justify the performance overhead. > > Frank Millman > > I thought Terry Reedy had shot down any claims about performance overhead, and that the memory savings in many cases must be substantial and therefore worthwhile. Or have I misread something? Or what? -- Cheers. Mark Lawrence.