Path: csiph.com!usenet.pasdenom.info!bete-des-vosges.org!news.redatomik.org!newsfeed.xs4all.nl!newsfeed1a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.001 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'encoding': 0.05; 'preferably': 0.05; 'say,': 0.05; 'subject:text': 0.05; 'reject': 0.07; 'utf-8': 0.07; 'bytes,': 0.09; 'subject:question': 0.10; 'cc:addr:python-list': 0.11; 'stored': 0.12; 'added.': 0.16; 'arbitrarily': 0.16; 'disallow': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'length,': 0.16; 'limiting': 0.16; 'merely': 0.16; 'pathological': 0.16; 'simple.': 0.16; 'surrogate': 0.16; 'truncate': 0.16; 'truncating': 0.16; 'two.': 0.16; 'username.': 0.16; 'prevent': 0.16; 'sat,': 0.16; 'wrote:': 0.18; 'putting': 0.22; 'cc:addr:python.org': 0.22; "aren't": 0.24; 'certainly': 0.24; 'logical': 0.24; 'string,': 0.24; 'unicode': 0.24; 'cc:2**0': 0.24; 'post': 0.26; 'header:In-Reply-To:1': 0.27; 'chris': 0.29; 'leave': 0.29; 'am,': 0.29; 'character': 0.29; 'points': 0.29; 'characters': 0.30; 'message-id:@mail.gmail.com': 0.30; 'work.': 0.31; 'code': 0.31; "user's": 0.31; 'probably': 0.32; 'beginning': 0.33; 'cases': 0.33; 'fri,': 0.33; 'problem': 0.35; "can't": 0.35; 'display': 0.35; 'point.': 0.35; "who's": 0.35; 'but': 0.35; 'received:google.com': 0.35; 'in.': 0.36; 'marks': 0.36; 'ones,': 0.36; 'half': 0.37; 'two': 0.37; 'whatever': 0.38; 'rather': 0.38; 'users': 0.40; 'course.': 0.60; 'then,': 0.60; 'most': 0.60; "you're": 0.61; 'first': 0.61; "you've": 0.63; 'name': 0.63; 'skip:n 10': 0.64; 'more': 0.64; 'mar': 0.68; 'combining': 0.68; 'limit': 0.70; 'to,': 0.72; '2015': 0.84; 'fifty': 0.84; 'points,': 0.84; 'to:none': 0.92 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-type; bh=GMmEP2cogxbiuzREmuuXxWooSQgyPxTWx6fQMSaOHSM=; b=YR5ml7pkkM8nvbYyoXfG7evyC31HQVcaF5pzHC0csS9omb1o7I2F4OoKq4YjtoH/My I8JZFKB4Obh36GI4J1I40m2bXUJvbB1I4DIRAzduRO6nOw3IMBwkzcLIFTaM36wE5dpI 5Na61h8ls7MIxNJkA/sz4cCSkWSmH7Jj2ZS8DuILQiFA4clkyQJYPy8JPN3cA4li8B4A u3MV9SK/iOz3QPtC1V6YDJUOE4s/muNuIsNAwTRNyYepZ0x6CWGF2thxOmLUAh6LWnrf JtctQCckFSLJ0gmE9Ojp85phX2nQosd6R2G6R6jnO8UDAsOOBJ0QpaaWGXjPg8oGiej/ vAWw== MIME-Version: 1.0 X-Received: by 10.50.4.40 with SMTP id h8mr54831859igh.34.1425651080730; Fri, 06 Mar 2015 06:11:20 -0800 (PST) In-Reply-To: <1425650591.3368169.236462481.4CA83873@webmail.messagingengine.com> References: <201502241524.t1OFO09k022270@fido.openend.se> <201502241620.t1OGKf4n002146@fido.openend.se> <54ECB134.5090304@davea.name> <201502241945.t1OJjshO013092@fido.openend.se> <201502241957.t1OJvrJS015604@fido.openend.se> <9169f3b1-2ac7-42a3-8033-584f84b88a1f@googlegroups.com> <7a75a23c-4678-4d7a-a2ec-9e8fff4c07f8@googlegroups.com> <132d5ce6-f672-4eec-99f9-1cc9e88b94f3@googlegroups.com> <619e4cb5-1c4c-449b-a5d7-951101b32b45@googlegroups.com> <54f862ca$0$13014$c3e8da3$5496439d@news.astraweb.com> <01dd9b83-db3e-4e7d-9022-dc6af75eb570@googlegroups.com> <1425648780.3358976.236451377.1ED1534A@webmail.messagingengine.com> <1425650591.3368169.236462481.4CA83873@webmail.messagingengine.com> Date: Sat, 7 Mar 2015 01:11:20 +1100 Subject: Re: Newbie question about text encoding From: Chris Angelico Cc: "python-list@python.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.19 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 45 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1425651082 news.xs4all.nl 2966 [2001:888:2000:d::a6]:40175 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:87024 On Sat, Mar 7, 2015 at 1:03 AM, wrote: > On Fri, Mar 6, 2015, at 08:39, Chris Angelico wrote: >> Number of code points is the most logical way to length-limit >> something. If you want to allow users to set their display names but >> not to make arbitrarily long ones, limiting them to X code points is >> the safest way (and preferably do an NFC or NFD normalization before >> counting, for consistency); > > Why are you length-limiting it? Storage space? Limit it in whatever > encoding they're stored in. Why are combining marks "pathological" but > surrogate characters not? Display space? Limit it by columns. If you're > going to allow a Japanese user's name to be twice as wide, you've got a > problem when you go to display it. To prevent people from putting three paragraphs of lipsum in and calling it a username. >> this means you disallow pathological cases >> where every base character has innumerable combining marks added. > > No it doesn't. If you limit it to, say, fifty, someone can still post > two base characters with twenty combining marks each. If you actually > want to disallow this, you've got to do more work. You've disallowed > some of the pathological cases, some of the time, by coincidence. And > limiting the number of UTF-8 bytes, or the number of UTF-16 code points, > will accomplish this just as well. They can, but then they're limited to two base characters. They can't have fifty base characters with twenty combining marks each. That's the point. > Now, if you intend to _silently truncate_ it to the desired length, you > certainly don't want to leave half a character in, of course. But who's > to say the base character plus first few combining marks aren't also > "half a character"? If you're _splitting_ a string, rather than merely > truncating it, you probably don't want those combining marks at the > beginning of part two. So you truncate to the desired length, then if the first character of the trimmed-off section is a combining mark (based on its Unicode character types), you keep trimming until you've removed a character which isn't. Then, if you no longer have any content whatsoever, reject the name. Simple. ChrisA