Path: csiph.com!usenet.pasdenom.info!news.redatomik.org!newsfeed.xs4all.nl!newsfeed4a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path: <techtonik@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.000
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'anyway.': 0.04; 'binary': 0.05; 'url:bitbucket': 0.05; '(first': 0.09; 'creighton': 0.09; 'from:addr:techtonik': 0.09; 'from:name:anatoly techtonik': 0.09; 'stdout': 0.09; 'stringio': 0.09; 'unicode,': 0.09; 'cc:addr :python-list': 0.10; 'python': 0.11; 'question.': 0.13; 'wed,': 0.15; 'all?': 0.16; 'anatoly': 0.16; 'appears.': 0.16; 'decode': 0.16; 'something.': 0.16; 'subject:unicode': 0.16; 'wrote:': 0.16; 'laura': 0.18; 'switched': 0.18; 'cc:2**0': 0.21; 'cc:addr:python.org': 0.21; 'explicit': 0.22; 'cc:no real name:2**0': 0.23; '2015': 0.23; 'header:In-Reply-To:1': 0.24; 'least': 0.27; 'message-id:@mail.gmail.com': 0.28; "doesn't": 0.28; 'windows,': 0.29; 'mode': 0.31; 'code': 0.31; 'advertised': 0.33; 'received:google.com': 0.34; 'fail': 0.35; 'unicode': 0.35; 'something': 0.35; 'really': 0.35; 'being': 0.36; 'text': 0.36; 'url:org': 0.36; '(and': 0.36; 'subject:: ': 0.37; 'missing': 0.37; 'stuff': 0.38; 'doing': 0.38; 'pm,': 0.39; 'data': 0.40; 'subject: (': 0.40; 'why': 0.40; 'your': 0.60; 'future': 0.61; 'information': 0.62; 'details': 0.63; 'here': 0.66; 'answer.': 0.72; 'obvious': 0.72
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=CHIbfkuzb6K8sT8MzP8WZ+437p21IMAdbN+NGWWTkK4=; b=DDNMKB1W+yT2XW7nIop4yjUsarD777eGvHwK64GIeqw9gpJPZxHTU7yNOUjjTggB3Z ohTCnhuuqu2IV/81cwVxLQfZr/t5P1vaUJsgNKtt4Yx2cl0sXCeC6dAgUA/oiNfHd1iq TY6xN9zLYi0eTxQ1LqZ/T2okhuCYQjw0QkbPvCXpstw/xW6CiEAq/vHR8wPM501XRRb9 ticjGOjpNeQWhaneO3Oq5hC6pUpcmJAR7rvleZ+VkrJT4oeOQiV8qw+oN18jMwQhXrFN mhNHJwMBUmgZRnQv8RBM2N3sHFtKhKBNebC7HtNm5SQeYJKjq82oOmm6/lg3WK7xnUYG DVqw==
X-Received: by 10.140.106.101 with SMTP id d92mr38753841qgf.70.1432727580546; Wed, 27 May 2015 04:53:00 -0700 (PDT)
MIME-Version: 1.0
In-Reply-To: <CAPkN8xJGnut_V0BmEah1ravN3cXESqx7c6U=mOAS0coD3fcPjA@mail.gmail.com>
References: <CAPkN8xKTXJu2nhvocG8KuyO1XkJVfK_WsmY6dM=hWsVyg+BVyA@mail.gmail.com> <techtonik@gmail.com> <CAPkN8xK452YWZvVRipTjq05eKthxEE28CB5W5-MN+6DZnXHOGg@mail.gmail.com> <201505271135.t4RBZY6F013836@fido.openend.se> <CAPkN8xJGnut_V0BmEah1ravN3cXESqx7c6U=mOAS0coD3fcPjA@mail.gmail.com>
From: anatoly techtonik <techtonik@gmail.com>
Date: Wed, 27 May 2015 14:52:40 +0300
Subject: Re: Fwd: Lossless bulletproof conversion to unicode (backslashing)
To: Laura Creighton <lac@openend.se>
Cc: python-list@python.org
Content-Type: text/plain; charset=UTF-8
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.20+
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.83.1432727585.5151.python-list@python.org>
Lines: 27
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1432727585 news.xs4all.nl 2914 [2001:888:2000:d::a6]:37935
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:91299

On Wed, May 27, 2015 at 2:47 PM, anatoly techtonik <techtonik@gmail.com> wrote:
> On Wed, May 27, 2015 at 2:35 PM, Laura Creighton <lac@openend.se> wrote:
>> I am missing something.  Why do you need unicode at all?  Why can you
>> not just keep your binary data as binary data?
>
> Good question. From the SCons code I see that we need unicode, because
> we switched to io.StringIO which is advertised as the future (and Python 3
> way of doing things, because Python 3 doesn't have non-unicode StringIO).
>
> A really deep and exhaustive answer.
> advertisement (first link on StringIO vs io.StringIO):
> http://stackoverflow.com/questions/3410309/what-is-the-difference-between-stringio-and-io-stringio-in-python2-7
> peaceful details
> https://bitbucket.org/scons/scons/commits/05d5af305a5d
> gory consequences
> https://bitbucket.org/scons/scons/pull-request/235/fix-tree-all-print-when-build-tree
>
>> I feel like I must be missing something obvious here ...
>
> Not that obvious as it appears.

And the short answer is that we need unicode because we are printing this
information to the stdout, and stdout is opened in text mode at least on
Windows, and without explicit conversion, Python will try to decode stuff
as being `ascii` and fail anyway.
-- 
anatoly t.