Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder1.enfer-du-nord.net!feeder.news-service.com!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path: <rosuav@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.001
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'oct': 0.02; 'encoded': 0.05; 'bytes.': 0.07; 'python': 0.08; 'encoding.': 0.09; 'subject:string': 0.09; 'utf-8': 0.09; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'hex': 0.16; 'string:': 0.16; 'subject:Unicode': 0.16; 'wrote:': 0.16; '>>>': 0.18; 'bytes': 0.18; 'received:209.85.210.174': 0.18; 'received:mail- iy0-f174.google.com': 0.18; 'assume': 0.22; 'header:In-Reply- To:1': 0.22; 'pm,': 0.24; 'byte': 0.24; 'string': 0.26; 'code,': 0.28; 'interpret': 0.29; 'unicode': 0.29; 'message- id:@mail.gmail.com': 0.29; 'strings,': 0.30; 'values': 0.32; 'represents': 0.32; 'does': 0.32; 'source': 0.33; 'probably': 0.33; 'to:addr:python-list': 0.33; 'instead': 0.33; 'reference': 0.35; 'johnson': 0.35; 'object': 0.35; 'fri,': 0.36; 'think': 0.38; 'received:google.com': 0.38; 'received:209.85': 0.38; 'subject:: ': 0.39; 'characters': 0.39; 'define': 0.39; 'to:addr:python.org': 0.39; 'your': 0.61; 'encoding,': 0.84
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=lv4HXg/ixDHR8GJokiRbF+Xi7Jd8ESM9U0YcPhvMBlw=; b=laRKobWHj+qNbHV0xvSSTUnPO85K4cUlJNkVcu0CMQK6umc+blmjm6+A2h5MG0vBfA TRPEyS3fzoFUc/Zhzcm+ouUymr+d8HyPJcidUk9enKHYwPw8w5gF30AcT2tRR5lepsMZ IdTJN4UGGQIj+WpI0tan0xWb4dmk+g7G2GHiU=
MIME-Version: 1.0
In-Reply-To: <8de0e9b7-aaac-44a4-8a07-9c1a20bfc3eb@s7g2000yqa.googlegroups.com>
References: <8de0e9b7-aaac-44a4-8a07-9c1a20bfc3eb@s7g2000yqa.googlegroups.com>
Date: Fri, 28 Oct 2011 14:38:39 +1100
Subject: Re: Unicode literals and byte string interpretation.
From: Chris Angelico <rosuav@gmail.com>
To: python-list@python.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.2273.1319773122.27778.python-list@python.org>
Lines: 24
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1319773122 news.xs4all.nl 6897 [2001:888:2000:d::a6]:51584
X-Complaints-To: abuse@xs4all.nl
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:15084

On Fri, Oct 28, 2011 at 2:05 PM, Fletcher Johnson <flt.johnson@gmail.com> w=
rote:
> If I create a new Unicode object u'\x82\xb1\x82\xea\x82\xcd' how does
> this creation process interpret the bytes in the byte string? Does it
> assume the string represents a utf-16 encoding, at utf-8 encoding,
> etc...?
>
> For reference the string is =E3=81=93=E3=82=8C=E3=81=AF in the 'shift-jis=
' encoding.

Encodings define how characters are represented in bytes. I think
probably what you're looking for is a byte string with those hex
values in it, which you can then turn into a Unicode string:

>>> a=3Db'\x82\xb1\x82\xea\x82\xcd'
>>> unicode(a,"shift-jis")    # use 'str' instead of 'unicode' in Python 3
u'\u3053\u308c\u306f'

The u'....' notation is for Unicode strings, which are not encoded in
any way. The last line of the above is a valid way of entering that
string in your source code, identifying Unicode characters by their
codepoints.

ChrisA