Path: csiph.com!x330-a1.tempe.blueboxinc.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!weretis.net!feeder1.news.weretis.net!feeder4.news.weretis.net!feeder.news-service.com!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=RNe067dICfcxgQrYzS+kvP7YZgVGjN8VbBg1/aPv6GjUZBc3ywa9Z7zgL6Tib8Lt3T NCbIriQqVcN6TkRw/UZXhnf9bNqbDgmaC552YhisFaDpDV5HmW7eBof9xAHz5Vvkz8o6 eUv/RT0UNbvNJA7wSuiP70AtsYf3l7iq8mwpo=
MIME-Version: 1.0
In-Reply-To: <4de546f8$0$6556$9b4e6d93@newsspool4.arcor-online.net>
References: <4de40ee8$0$6623$9b4e6d93@newsspool2.arcor-online.net> <mailman.2315.1306841548.9059.python-list@python.org> <4de50cfd$0$6538$9b4e6d93@newsspool4.arcor-online.net> <4de546f8$0$6556$9b4e6d93@newsspool4.arcor-online.net>
Date: Wed, 1 Jun 2011 07:56:24 +1000
Subject: Re: Thanks for all responses
From: Chris Angelico <rosuav@gmail.com>
To: python-list@python.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.2337.1306878988.9059.python-list@python.org>
Lines: 90
NNTP-Posting-Host: 82.94.164.166
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:6759

On Wed, Jun 1, 2011 at 5:52 AM, Wolfgang Meiners
<WolfgangMeiners01@web.de> wrote:
> Whenever i 'cross the border' of my program, i have to encode the 'list
> of bytes' to an unicode string or decode the unicode string to a 'list
> of bytes' which is meaningful to the world outside.

Most people use "encode" and "decode" the other way around; you encode
a string as UTF-8, and decode UTF-8 into a Unicode string. But yes,
you're correct.

> So encode early, decode lately means, to do it as near to the border as
> possible and to encode/decode i need a coding system, for example 'utf8'

Correct on both counts.

> That means, there should be an encoding/decoding possibility to every
> interface i can use: files, stdin, stdout, stderr, gui (should be the
> most important ones).

The file objects (as returned by open()) have an encoding, which
(IMHO) defaults to "utf8". GUI work depends on your GUI toolkit, and
might well accept Unicode strings directly - check the docs.

> =A0 =A0def __repr__(self):
> =A0 =A0 =A0 =A0return u'My name is %s' % self.Name

This means that repr() will return a Unicode string.

> =A0 =A0# this does work
> =A0 =A0print a.__repr__()
>
> =A0 =A0# throws an error if default encoding is ascii
> =A0 =A0# but works if default encoding is utf8
> =A0 =A0print a
>
> =A0 =A0# throws an error because a is not a string
> =A0 =A0print unicode(a, encoding=3D'utf8')

The __repr__ function is supposed to return a string object, in Python
2. See http://docs.python.org/reference/datamodel.html#object.__repr__
for that and other advice on writing __repr__. The problems you're
seeing are a result of the built-in repr() function calling
a.__repr__() and then treating the return value as an ASCII str, not a
Unicode string.

This would work:
 =A0 =A0def __repr__(self):
 =A0 =A0 =A0 =A0return (u'My name is %s' % self.Name).encode('utf8')

Alternatively, migrate to Python 3, where the default is Unicode
strings. I tested this in Python 3.2 on Windows, but it should work on
anything in the 3.x branch:

class NoEnc:
	def __init__(self,Name=3DNone):
		self.Name=3DName
	def __repr__(self):
		return 'My name is %s' % self.Name

if __name__ =3D=3D '__main__':

   a =3D NoEnc('M=FCller')

   # this will still work (print is now a function, not a statement)
   print(a.__repr__())

   # this will work in Python 3.x
   print(a)

   # 'unicode' has been renamed to 'str', but it's already unicode so
this makes no sense
   print(str(a, encoding=3D'utf8'))

   # to convert it to UTF-8, convert it to a string with str() or
repr() and then print:
   print(str(a).encode('utf8'))
############################

Note that the last one will probably not do what you expect. The
Python 3 'print' function (it's not a statement any more, so you need
parentheses around its argument) wants a Unicode string, so you don't
need to encode it. When you encode a Unicode string as in the last
example, it returns a bytes string (an array of bytes), which looks
like this: b'My name is M\xc3\xbcller'  The print function wants
Unicode, though, so it takes this unexpected object and calls str() on
it, hence the odd display.

Hope that helps!

Chris Angelico