Re: Fwd: Lossless bulletproof conversion to unicode (backslashing)

Path	csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!1.eu.feeder.erje.net!ecngs!feeder2.ecngs.de!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path	<rosuav@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.004
X-Spam-Evidence	'H': 0.99; 'S': 0.00; 'anyway.': 0.04; 'bytes.': 0.07; 'rest,': 0.07; 'stdout': 0.09; 'cc:addr:python-list': 0.10; 'python': 0.11; 'question.': 0.13; 'wed,': 0.15; 'anatoly': 0.16; 'decode': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'subject:unicode': 0.16; 'wrote:': 0.16; 'bytes': 0.18; 'cc:2**0': 0.21; 'cc:addr:python.org': 0.21; 'explicit': 0.22; '2015': 0.23; 'header:In-Reply-To:1': 0.24; 'least': 0.27; 'message-id:@mail.gmail.com': 0.28; 'windows,': 0.29; 'no,': 0.29; 'mode': 0.31; 'print': 0.31; "can't": 0.32; 'received:google.com': 0.34; 'fail': 0.35; 'text.': 0.35; 'unicode': 0.35; 'list': 0.35; 'being': 0.36; 'text': 0.36; 'subject:: ': 0.37; "won't": 0.38; 'stuff': 0.38; 'means': 0.39; 'pm,': 0.39; 'subject: (': 0.40; 'information': 0.62; 'you.': 0.64; 'chrisa': 0.84; 'everything,': 0.84; 'to:none': 0.90
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-type; bh=ZQ7HhkmlqJxAw2h+QiyMgHKUQAFtkANkD3qS3t0d5mA=; b=XDnR0d1WKe5QTVXcvLGM96zgPVVac/0MCT+WExECkk5pSztjmRzHX1nTUSZqhEeuN6 fZjhKTOJfYRNxN/Kt8r+WVvMK+BrlAWYDR0FHZsofK7qw+PSRxekeNHdWqCcFNm4+/5g 0H0g8yOoWIalOv3x4XmFpMNrxt0LBo+PIdYZKZB8qTQukRZfKONnCLAJM7//XbD5A+TQ 0IYzOJQ/rPVEm0xY0qFvv+uOFcreu7z7gOvUyujwYRcwfzkd8oBwNA/kGd+QcjFZ96H2 WTlCuORkraRmgUORNh3tXGE87POQUQArNn5bgjEd5Dw9cYCt/yW2NBCEEK//10ebNq4v v7gg==
MIME-Version	1.0
X-Received	by 10.42.43.199 with SMTP id y7mr2910171ice.12.1432727943610; Wed, 27 May 2015 04:59:03 -0700 (PDT)
In-Reply-To	<CAPkN8xJTA_cfWuXD8+MUqKB6yTMp=vzUHk0sQsNz3acyJL-LXg@mail.gmail.com>
References	<CAPkN8xKTXJu2nhvocG8KuyO1XkJVfK_WsmY6dM=hWsVyg+BVyA@mail.gmail.com> <techtonik@gmail.com> <CAPkN8xK452YWZvVRipTjq05eKthxEE28CB5W5-MN+6DZnXHOGg@mail.gmail.com> <201505271135.t4RBZY6F013836@fido.openend.se> <CAPkN8xJGnut_V0BmEah1ravN3cXESqx7c6U=mOAS0coD3fcPjA@mail.gmail.com> <CAPkN8xJTA_cfWuXD8+MUqKB6yTMp=vzUHk0sQsNz3acyJL-LXg@mail.gmail.com>
Date	Wed, 27 May 2015 21:59:03 +1000
Subject	Re: Fwd: Lossless bulletproof conversion to unicode (backslashing)
From	Chris Angelico <rosuav@gmail.com>
Cc	"python-list@python.org" <python-list@python.org>
Content-Type	text/plain; charset=UTF-8
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.20+
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.84.1432727951.5151.python-list@python.org> (permalink)
Lines	16
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1432727951 news.xs4all.nl 2856 [2001:888:2000:d::a6]:43203
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:91300

Show key headers only | View raw

On Wed, May 27, 2015 at 9:52 PM, anatoly techtonik <techtonik@gmail.com> wrote:
> And the short answer is that we need unicode because we are printing this
> information to the stdout, and stdout is opened in text mode at least on
> Windows, and without explicit conversion, Python will try to decode stuff
> as being `ascii` and fail anyway.

So you're working with text. That means you HAVE to decode it somehow;
you fundamentally cannot print bytes to the console. Lossless
concealment of arbitrary bytes won't help you. If you can't adequately
decode everything, either backslash-escape the rest, or use a
replacement character; you can't print out those bytes.

And no, I will not cc you. Subscribe to the list if you're going to
ask a question.

ChrisA

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: Fwd: Lossless bulletproof conversion to unicode (backslashing) Chris Angelico <rosuav@gmail.com> - 2015-05-27 21:59 +1000

csiph-web