Path: csiph.com!usenet.pasdenom.info!gegeweb.org!usenet-fr.net!nerim.net!novso.com!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path: <random832@fastmail.us>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.003
X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'string.': 0.05; '(especially': 0.07; 'subject:file': 0.07; 'string': 0.09; 'received:internal': 0.09; 'encodings': 0.16; 'merely': 0.16; 'message-id:@webmail.messagingengine.com': 0.16; 'received:10.202': 0.16; 'received:10.202.2': 0.16; 'received:66.111': 0.16; 'received:66.111.4': 0.16; 'received:66.111.4.27': 0.16; 'received:messagingengine.com': 0.16; 'received:out3-smtp.messagingengine.com': 0.16; 'storing': 0.16; 'subject:String': 0.16; 'windows?': 0.16; 'wrote:': 0.18; 'pfxlen:0': 0.19; 'split': 0.19; '(the': 0.22; 'unicode': 0.24; 'fine': 0.24; 'source': 0.25; 'van': 0.27; 'header:In-Reply-To:1': 0.27; 'to:2**1': 0.27; '----': 0.29; 'array': 0.29; 'character': 0.29; 'points': 0.29; "doesn't": 0.30; 'characters': 0.30; "i'm": 0.30; 'code': 0.31; 'ctypes': 0.31; 'sep': 0.31; 'sets.': 0.31; 'this.': 0.32; 'fri,': 0.33; 'subject:the': 0.34; 'could': 0.34; 'received:66': 0.35; 'objects': 0.35; 'but': 0.35; 'described': 0.36; 'interact': 0.36; 'received:10': 0.37; 'problems': 0.38; 'to:addr:python-list': 0.38; 'rather': 0.38; 'does': 0.39; 'sure': 0.39; 'to:addr:python.org': 0.39; 'how': 0.40; 'skip:u 10': 0.60; 'solve': 0.60; 'from:no real name:2**0': 0.61; 'simply': 0.61; 'header:Message-Id:1': 0.63; 'different': 0.65; 'therefore': 0.72; '2013,': 0.91
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=fastmail.us; h= message-id:from:to:mime-version:content-transfer-encoding :content-type:subject:date:in-reply-to:references; s=mesmtp; bh= kmzsB4QW0jOauU1j376OkAz6Qcc=; b=e1uFXIC9ceSYowXYMLYI7XXJ0Q7E9DT5 J+yxFteKOg9Z4VVnCYnR7VtdwO1VBs/XIPGggHJAJvmRE4e19d1z9rNpBGtp0wGQ GkKoGnBRbOOsNVM64hEyxsMpb0X7GsvWb8oWFb2OMyFJSi/AS0LR3iJtSZfzg3lR CP1GJEsGECs=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=message-id:from:to:mime-version :content-transfer-encoding:content-type:subject:date:in-reply-to :references; s=smtpout; bh=kmzsB4QW0jOauU1j376OkAz6Qcc=; b=bxnwh Z6zvA78cDkH8biFuOGHi2zdRSd7LhbpLqfu0pzFyMzAFMl4Xs3+yfzQnOAnqJ7Md BOE/MYMG9JLXWaip0nKcSsVfczxzA99SHYSR2WDzr49vjrYLQ3JD+6DSS3IFr1re hz2zhy+8FD1L1R+VRCkJf5U7DSNVQp7Mv/mvQE=
X-Sasl-Enc: Zg3b/Hiw4weysVSRat2Sv6I8Gx7nfYn+mnEQJwfDZuQ6 1378486748
From: random832@fastmail.us
To: Piet van Oostrum <piet@vanoostrum.org>, python-list@python.org
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain
X-Mailer: MessagingEngine.com Webmail Interface - ajax-22680048
Subject: Re: Chardet, file, ... and the Flexible String Representation
Date: Fri, 06 Sep 2013 12:59:08 -0400
In-Reply-To: <m2a9jqq7g9.fsf@cochabamba.vanoostrum.org>
References: <4ce85ea8-4a4c-46cf-a546-ad999576a5f7@googlegroups.com> <m2a9jqq7g9.fsf@cochabamba.vanoostrum.org>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.128.1378486750.5461.python-list@python.org>
Lines: 21
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1378486750 news.xs4all.nl 16009 [2001:888:2000:d::a6]:42990
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:53796

On Fri, Sep 6, 2013, at 11:46, Piet van Oostrum wrote:
> The FSR does not split unicode in chuncks. It does not create problems
> and therefore it doesn't have to solve this. 
> 
> The FSR simply stores a Unicode string as an array[*] of ints (the
> Unicode code points of the characters of the string. That's it. Then it
> uses a memory-efficient way to store this array of ints. But that has
> nothing to do with character sets. The same principle could be used for
> any array of ints.

I think the source of the confusion is that it is described in terms of
UCS-2 and Latin-1, which people often think of (especially latin-1) as
different encodings rather than merely storing code points in a narrower
type.

----

Incidentally, how does all this interact with ctypes unicode_buffers,
which slice as strings and must be UTF-16 on windows? This was fine
pre-FSR when unicode objects were UTF-16, but I'm not sure how it would
work now.