Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!eternal-september.org!feeder.eternal-september.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.001 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'encoded': 0.05; 'result,': 0.05; 'matches': 0.07; 'strings.': 0.07; '*is*': 0.09; '128': 0.09; 'effect.': 0.09; 'pep': 0.09; 'subject:string': 0.09; 'unicode,': 0.09; 'cc:addr:python-list': 0.10; '"code': 0.16; 'cp1252': 0.16; 'enough.': 0.16; 'example)': 0.16; 'mistake.': 0.16; 'subject:3.3': 0.16; '\xe9crit': 0.16; 'string': 0.17; 'wrote:': 0.17; 'byte': 0.17; 'bytes': 0.17; 'unicode': 0.17; 'code.': 0.20; 'trying': 0.21; 'bit': 0.21; 'work,': 0.22; 'cc:2**0': 0.23; 'cc:no real name:2**0': 0.24; 'second': 0.24; 'cc:addr:python.org': 0.25; 'header:In-Reply-To:1': 0.25; 'header :User-Agent:1': 0.26; 'common': 0.26; 'ago': 0.27; 'am,': 0.27; 'coding': 0.27; 'converting': 0.27; 'correct': 0.28; 'chris': 0.28; 'accidentally': 0.29; 'character.': 0.29; 'description,': 0.29; 'lies': 0.29; 'omitted': 0.29; 'scheme.': 0.29; 'character': 0.29; 'no,': 0.29; 'points': 0.29; 'push': 0.30; 'basic': 0.30; 'sense': 0.31; 'code': 0.31; 'european': 0.33; 'problem': 0.33; 'point.': 0.33; 'largest': 0.35; 'said,': 0.35; 'but': 0.36; 'expensive': 0.36; 'should': 0.36; 'uses': 0.37; '(for': 0.37; 'subject:New': 0.37; 'rather': 0.37; 'far': 0.37; 'subject:: ': 0.38; 'store': 0.38; 'fact': 0.38; 'several': 0.39; 'received:192': 0.39; 'received:192.168': 0.40; 'think': 0.40; 'your': 0.60; 'most': 0.61; 'side': 0.61; 'kind': 0.61; 'email addr:gmail.com': 0.63; 'more': 0.63; 'header:Reply-To:1': 0.68; 'received:74.208': 0.71; 'reply-to:no real name:2**0': 0.72; "it'd": 0.84; 'words)': 0.84; 'effects.': 0.91 Date: Sun, 19 Aug 2012 08:35:23 -0400 From: Dave Angel User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20120714 Thunderbird/14.0 MIME-Version: 1.0 To: wxjmfauth@gmail.com Subject: Re: New internal string format in 3.3 References: <308df2af-abe7-4043-b199-0a39f440e0ab@googlegroups.com> <502f8a2a$0$29978$c3e8da3$5496439d@news.astraweb.com> <4c62a649-bc21-4e47-9c0f-acb1b1e70e36@googlegroups.com> <5030891f$0$29978$c3e8da3$5496439d@news.astraweb.com> <5030aa44$0$29978$c3e8da3$5496439d@news.astraweb.com> <11931ec9-1858-4ae8-8a61-1d154d105229@googlegroups.com> <73c85f3b-a4a9-4812-bc41-132b5126874c@googlegroups.com> In-Reply-To: <73c85f3b-a4a9-4812-bc41-132b5126874c@googlegroups.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Provags-ID: V02:K0:ccPxBrIelipBaKY1Wm83t2qAQSABKFQ/jXaR8WFcM5j DhHqmAjcG4j2KXngOgqP72MuHKrE0k7ahXuvYv6+tlj4tYV2+n EYm1JuhKTQkbQEFSYIpIMQYpKPyQGcQ85W+RcuxdE4rAcJvLM2 2V1e1sKtRyyymIcJXPoLGEer2ihDKceix0PNF5ZwVzR+IPUjrP pykAMzIW8CU98fw7pMNHoW543atx71soq9algHt1qw69NPoW7k d1L7Iku+vkKTet2pwhaJOAb65RL3NfuU6Elo2mvERB6wDvXc/Y HkknwsAAyzau8yO9Ri77rD135NjWx0dAKSKjMZ2cotgi/WsRw= = Cc: python-list@python.org X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: d@davea.name List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 39 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1345379751 news.xs4all.nl 6884 [2001:888:2000:d::a6]:39673 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:27385 (pardon the resend, but I accidentally omitted a couple of words) On 08/19/2012 08:14 AM, wxjmfauth@gmail.com wrote: > Le dimanche 19 août 2012 12:26:44 UTC+2, Chris Angelico a écrit : >> >> >> >> No, it uses Unicode, and as an optimization, attempts to store the >> codepoints in less than four bytes for most strings. The fact that a >> one-byte storage format happens to look like latin-1 is rather >> coincidental. >> > And this this is the common basic mistake. You do not push your > argumentation far enough. A character may "fall" accidentally in a latin-1. > The problem lies in these european characters, which can not fall in this > coding. This *is* the cause of the negative side effects. > If you are using a correct coding scheme, like cp1252, mac-roman or > iso-8859-15, you will never see such a negative side effect. > Again, the problem is not the result, the encoded character. The critical > part is the character which may cause this side effect. > You should think "character set" and not encoded "code point", considering > this kind of expression has a sense in 8-bits coding scheme. > > jmf But that choice was made decades ago when Unicode picked its second 128 characters. The internal form used in this PEP is simply the low-order byte of the Unicode code point. Trying to scan the string deciding if converting to cp1252 (for example) would work, would be a much more expensive operation than seeing how many bytes it'd take for the largest code point. The 8 bit form is used if all the code points are less than 256. That is a simple description, and simple code. As several people have said, the fact that this byte matches on of the DECODED forms is coincidence. -- DaveA