Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!1.eu.feeder.erje.net!ecngs!feeder2.ecngs.de!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path: <rosuav@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.004
X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'anyway.': 0.04; 'bytes.': 0.07; 'rest,': 0.07; 'stdout': 0.09; 'cc:addr:python-list': 0.10; 'python': 0.11; 'question.': 0.13; 'wed,': 0.15; 'anatoly': 0.16; 'decode': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'subject:unicode': 0.16; 'wrote:': 0.16; 'bytes': 0.18; 'cc:2**0': 0.21; 'cc:addr:python.org': 0.21; 'explicit': 0.22; '2015': 0.23; 'header:In-Reply-To:1': 0.24; 'least': 0.27; 'message-id:@mail.gmail.com': 0.28; 'windows,': 0.29; 'no,': 0.29; 'mode': 0.31; 'print': 0.31; "can't": 0.32; 'received:google.com': 0.34; 'fail': 0.35; 'text.': 0.35; 'unicode': 0.35; 'list': 0.35; 'being': 0.36; 'text': 0.36; 'subject:: ': 0.37; "won't": 0.38; 'stuff': 0.38; 'means': 0.39; 'pm,': 0.39; 'subject: (': 0.40; 'information': 0.62; 'you.': 0.64; 'chrisa': 0.84; 'everything,': 0.84; 'to:none': 0.90
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-type; bh=ZQ7HhkmlqJxAw2h+QiyMgHKUQAFtkANkD3qS3t0d5mA=; b=XDnR0d1WKe5QTVXcvLGM96zgPVVac/0MCT+WExECkk5pSztjmRzHX1nTUSZqhEeuN6 fZjhKTOJfYRNxN/Kt8r+WVvMK+BrlAWYDR0FHZsofK7qw+PSRxekeNHdWqCcFNm4+/5g 0H0g8yOoWIalOv3x4XmFpMNrxt0LBo+PIdYZKZB8qTQukRZfKONnCLAJM7//XbD5A+TQ 0IYzOJQ/rPVEm0xY0qFvv+uOFcreu7z7gOvUyujwYRcwfzkd8oBwNA/kGd+QcjFZ96H2 WTlCuORkraRmgUORNh3tXGE87POQUQArNn5bgjEd5Dw9cYCt/yW2NBCEEK//10ebNq4v v7gg==
MIME-Version: 1.0
X-Received: by 10.42.43.199 with SMTP id y7mr2910171ice.12.1432727943610; Wed, 27 May 2015 04:59:03 -0700 (PDT)
In-Reply-To: <CAPkN8xJTA_cfWuXD8+MUqKB6yTMp=vzUHk0sQsNz3acyJL-LXg@mail.gmail.com>
References: <CAPkN8xKTXJu2nhvocG8KuyO1XkJVfK_WsmY6dM=hWsVyg+BVyA@mail.gmail.com> <techtonik@gmail.com> <CAPkN8xK452YWZvVRipTjq05eKthxEE28CB5W5-MN+6DZnXHOGg@mail.gmail.com> <201505271135.t4RBZY6F013836@fido.openend.se> <CAPkN8xJGnut_V0BmEah1ravN3cXESqx7c6U=mOAS0coD3fcPjA@mail.gmail.com> <CAPkN8xJTA_cfWuXD8+MUqKB6yTMp=vzUHk0sQsNz3acyJL-LXg@mail.gmail.com>
Date: Wed, 27 May 2015 21:59:03 +1000
Subject: Re: Fwd: Lossless bulletproof conversion to unicode (backslashing)
From: Chris Angelico <rosuav@gmail.com>
Cc: "python-list@python.org" <python-list@python.org>
Content-Type: text/plain; charset=UTF-8
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.20+
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.84.1432727951.5151.python-list@python.org>
Lines: 16
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1432727951 news.xs4all.nl 2856 [2001:888:2000:d::a6]:43203
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:91300

On Wed, May 27, 2015 at 9:52 PM, anatoly techtonik <techtonik@gmail.com> wrote:
> And the short answer is that we need unicode because we are printing this
> information to the stdout, and stdout is opened in text mode at least on
> Windows, and without explicit conversion, Python will try to decode stuff
> as being `ascii` and fail anyway.

So you're working with text. That means you HAVE to decode it somehow;
you fundamentally cannot print bytes to the console. Lossless
concealment of arbitrary bytes won't help you. If you can't adequately
decode everything, either backslash-escape the rest, or use a
replacement character; you can't print out those bytes.

And no, I will not cc you. Subscribe to the list if you're going to
ask a question.

ChrisA