Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #15080 > unrolled thread

Unicode literals and byte string interpretation.

Back to article view | Back to comp.lang.python

Contents

  Unicode literals and byte string interpretation. Fletcher Johnson <flt.johnson@gmail.com> - 2011-10-27 20:05 -0700
    Re: Unicode literals and byte string interpretation. David Riley <fraveydank@gmail.com> - 2011-10-27 23:37 -0400
    Re: Unicode literals and byte string interpretation. Chris Angelico <rosuav@gmail.com> - 2011-10-28 14:38 +1100
    Re: Unicode literals and byte string interpretation. Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-10-28 07:06 +0000
      Re: Unicode literals and byte string interpretation. Fletcher Johnson <flt.johnson@gmail.com> - 2011-10-31 21:01 -0700

#15080 — Unicode literals and byte string interpretation.

If I create a new Unicode object u'\x82\xb1\x82\xea\x82\xcd' how does
this creation process interpret the bytes in the byte string? Does it
assume the string represents a utf-16 encoding, at utf-8 encoding,
etc...?

For reference the string is これは in the 'shift-jis' encoding.

[toc] | [next] | [standalone]

#15083

On Oct 27, 2011, at 11:05 PM, Fletcher Johnson wrote:

> If I create a new Unicode object u'\x82\xb1\x82\xea\x82\xcd' how does
> this creation process interpret the bytes in the byte string? Does it
> assume the string represents a utf-16 encoding, at utf-8 encoding,
> etc...?
> 
> For reference the string is これは in the 'shift-jis' encoding.

Try it and see!  One test case is worth a thousand words.  And Python has an interactive interpreter. :-)


- Dave

[toc] | [prev] | [next] | [standalone]

#15084

On Fri, Oct 28, 2011 at 2:05 PM, Fletcher Johnson <flt.johnson@gmail.com> wrote:
> If I create a new Unicode object u'\x82\xb1\x82\xea\x82\xcd' how does
> this creation process interpret the bytes in the byte string? Does it
> assume the string represents a utf-16 encoding, at utf-8 encoding,
> etc...?
>
> For reference the string is これは in the 'shift-jis' encoding.

Encodings define how characters are represented in bytes. I think
probably what you're looking for is a byte string with those hex
values in it, which you can then turn into a Unicode string:

>>> a=b'\x82\xb1\x82\xea\x82\xcd'
>>> unicode(a,"shift-jis")    # use 'str' instead of 'unicode' in Python 3
u'\u3053\u308c\u306f'

The u'....' notation is for Unicode strings, which are not encoded in
any way. The last line of the above is a valid way of entering that
string in your source code, identifying Unicode characters by their
codepoints.

ChrisA

[toc] | [prev] | [next] | [standalone]

#15090

On Thu, 27 Oct 2011 20:05:13 -0700, Fletcher Johnson wrote:

> If I create a new Unicode object u'\x82\xb1\x82\xea\x82\xcd' how does
> this creation process interpret the bytes in the byte string? 

It doesn't, because there is no byte-string. You have created a Unicode 
object from a literal string of unicode characters, not bytes. Those 
characters are:

Dec Hex  Char
130 0x82 ‚
177 0xb1 ±
130 0x82 ‚
234 0xea ê
130 0x82 ‚
205 0xcd Í

Don't be fooled that all of the characters happen to be in the range 
0-255, that is irrelevant.


> Does it
> assume the string represents a utf-16 encoding, at utf-8 encoding,
> etc...?

None of the above. It assumes nothing. It takes a string of characters, 
end of story.

> For reference the string is これは in the 'shift-jis' encoding.

No it is not. The way to get a unicode literal with those characters is 
to use a unicode-aware editor or terminal:

>>> s = u'これは'
>>> for c in s:
...     print ord(c), hex(ord(c)), c
... 
12371 0x3053 こ
12428 0x308c れ
12399 0x306f は


You are confusing characters with bytes. I believe that what you are 
thinking of is the following: you start with a byte string, and then 
decode it into unicode:

>>> bytes = '\x82\xb1\x82\xea\x82\xcd'  # not u'...'
>>> text = bytes.decode('shift-jis')
>>> print text
これは


If you get the encoding wrong, you will get the wrong characters:

>>> print bytes.decode('utf-16')
놂춂


If you start with the Unicode characters, you can encode it into various 
byte strings:

>>> s = u'これは'
>>> s.encode('shift-jis')
'\x82\xb1\x82\xea\x82\xcd'
>>> s.encode('utf-8')
'\xe3\x81\x93\xe3\x82\x8c\xe3\x81\xaf'






-- 
Steven

[toc] | [prev] | [next] | [standalone]

#15197

On Oct 28, 3:06 am, Steven D'Aprano <steve
+comp.lang.pyt...@pearwood.info> wrote:
> On Thu, 27 Oct 2011 20:05:13 -0700, Fletcher Johnson wrote:
> > If I create a newUnicodeobject u'\x82\xb1\x82\xea\x82\xcd' how does
> > this creation process interpret the bytes in the byte string?
>
> It doesn't, because there is no byte-string. You have created aUnicode
> object from aliteralstring ofunicodecharacters, not bytes. Those
> characters are:
>
> Dec Hex  Char
> 130 0x82 ‚
> 177 0xb1 ±
> 130 0x82 ‚
> 234 0xea ê
> 130 0x82 ‚
> 205 0xcd Í
>
> Don't be fooled that all of the characters happen to be in the range
> 0-255, that is irrelevant.
>
> > Does it
> > assume the string represents a utf-16 encoding, at utf-8 encoding,
> > etc...?
>
> None of the above. It assumes nothing. It takes a string of characters,
> end of story.
>
> > For reference the string is これは in the 'shift-jis' encoding.
>
> No it is not. The way to get aunicodeliteralwith those characters is
> to use aunicode-aware editor or terminal:
>
> >>> s = u'これは'
> >>> for c in s:
>
> ...     print ord(c), hex(ord(c)), c
> ...
> 12371 0x3053 こ
> 12428 0x308c れ
> 12399 0x306f は
>
> You are confusing characters with bytes. I believe that what you are
> thinking of is the following: you start with a byte string, and then
> decode it intounicode:
>
> >>> bytes = '\x82\xb1\x82\xea\x82\xcd'  # not u'...'
> >>> text = bytes.decode('shift-jis')
> >>> print text
>
> これは
>
> If you get the encoding wrong, you will get the wrong characters:
>
> >>> print bytes.decode('utf-16')
>
> 놂춂
>
> If you start with theUnicodecharacters, you can encode it into various
> byte strings:
>
> >>> s = u'これは'
> >>> s.encode('shift-jis')
>
> '\x82\xb1\x82\xea\x82\xcd'>>> s.encode('utf-8')
>
> '\xe3\x81\x93\xe3\x82\x8c\xe3\x81\xaf'
>
> --
> Steven

Thanks Steven. You are right. I was confusing characters with bytes.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web