Re: Unicode literals and byte string interpretation.

Path	csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder1.enfer-du-nord.net!feeder.news-service.com!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path	<rosuav@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.001
X-Spam-Evidence	'H': 1.00; 'S': 0.00; 'oct': 0.02; 'encoded': 0.05; 'bytes.': 0.07; 'python': 0.08; 'encoding.': 0.09; 'subject:string': 0.09; 'utf-8': 0.09; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'hex': 0.16; 'string:': 0.16; 'subject:Unicode': 0.16; 'wrote:': 0.16; '>>>': 0.18; 'bytes': 0.18; 'received:209.85.210.174': 0.18; 'received:mail- iy0-f174.google.com': 0.18; 'assume': 0.22; 'header:In-Reply- To:1': 0.22; 'pm,': 0.24; 'byte': 0.24; 'string': 0.26; 'code,': 0.28; 'interpret': 0.29; 'unicode': 0.29; 'message- id:@mail.gmail.com': 0.29; 'strings,': 0.30; 'values': 0.32; 'represents': 0.32; 'does': 0.32; 'source': 0.33; 'probably': 0.33; 'to:addr:python-list': 0.33; 'instead': 0.33; 'reference': 0.35; 'johnson': 0.35; 'object': 0.35; 'fri,': 0.36; 'think': 0.38; 'received:google.com': 0.38; 'received:209.85': 0.38; 'subject:: ': 0.39; 'characters': 0.39; 'define': 0.39; 'to:addr:python.org': 0.39; 'your': 0.61; 'encoding,': 0.84
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=lv4HXg/ixDHR8GJokiRbF+Xi7Jd8ESM9U0YcPhvMBlw=; b=laRKobWHj+qNbHV0xvSSTUnPO85K4cUlJNkVcu0CMQK6umc+blmjm6+A2h5MG0vBfA TRPEyS3fzoFUc/Zhzcm+ouUymr+d8HyPJcidUk9enKHYwPw8w5gF30AcT2tRR5lepsMZ IdTJN4UGGQIj+WpI0tan0xWb4dmk+g7G2GHiU=
MIME-Version	1.0
In-Reply-To	<8de0e9b7-aaac-44a4-8a07-9c1a20bfc3eb@s7g2000yqa.googlegroups.com>
References	<8de0e9b7-aaac-44a4-8a07-9c1a20bfc3eb@s7g2000yqa.googlegroups.com>
Date	Fri, 28 Oct 2011 14:38:39 +1100
Subject	Re: Unicode literals and byte string interpretation.
From	Chris Angelico <rosuav@gmail.com>
To	python-list@python.org
Content-Type	text/plain; charset=UTF-8
Content-Transfer-Encoding	quoted-printable
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.12
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.2273.1319773122.27778.python-list@python.org> (permalink)
Lines	24
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1319773122 news.xs4all.nl 6897 [2001:888:2000:d::a6]:51584
X-Complaints-To	abuse@xs4all.nl
Xref	x330-a1.tempe.blueboxinc.net comp.lang.python:15084

Show key headers only | View raw

On Fri, Oct 28, 2011 at 2:05 PM, Fletcher Johnson <flt.johnson@gmail.com> wrote:
> If I create a new Unicode object u'\x82\xb1\x82\xea\x82\xcd' how does
> this creation process interpret the bytes in the byte string? Does it
> assume the string represents a utf-16 encoding, at utf-8 encoding,
> etc...?
>
> For reference the string is これは in the 'shift-jis' encoding.

Encodings define how characters are represented in bytes. I think
probably what you're looking for is a byte string with those hex
values in it, which you can then turn into a Unicode string:

>>> a=b'\x82\xb1\x82\xea\x82\xcd'
>>> unicode(a,"shift-jis")    # use 'str' instead of 'unicode' in Python 3
u'\u3053\u308c\u306f'

The u'....' notation is for Unicode strings, which are not encoded in
any way. The last line of the above is a valid way of entering that
string in your source code, identifying Unicode characters by their
codepoints.

ChrisA

Thread

Unicode literals and byte string interpretation. Fletcher Johnson <flt.johnson@gmail.com> - 2011-10-27 20:05 -0700
  Re: Unicode literals and byte string interpretation. David Riley <fraveydank@gmail.com> - 2011-10-27 23:37 -0400
  Re: Unicode literals and byte string interpretation. Chris Angelico <rosuav@gmail.com> - 2011-10-28 14:38 +1100
  Re: Unicode literals and byte string interpretation. Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-10-28 07:06 +0000
    Re: Unicode literals and byte string interpretation. Fletcher Johnson <flt.johnson@gmail.com> - 2011-10-31 21:01 -0700

csiph-web