Path: csiph.com!x330-a1.tempe.blueboxinc.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!weretis.net!feeder1.news.weretis.net!news.albasani.net!news2.arglkargh.de!news.wiretrip.org!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path: <rosuav@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.001
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'wed,': 0.03; '(of': 0.05; 'bytes.': 0.07; 'defines': 0.07; 'used.': 0.07; 'python': 0.08; 'bytes,': 0.09; 'many)': 0.09; 'utf-8': 0.09; 'am,': 0.14; 'wrote:': 0.14; '>>and': 0.16; 'angelico': 0.16; 'dictionaries': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'mythical': 0.16; 'standard)': 0.16; 'subject:Unicode': 0.16; 'converting': 0.16; 'convert': 0.19; 'header:In-Reply-To:1': 0.21; 'primarily': 0.22; 'memory': 0.22; 'received:209.85.210.174': 0.23; 'received:mail-iy0-f174.google.com': 0.23; 'works.': 0.23; 'stored': 0.25; 'message-id:@mail.gmail.com': 0.28; '(the': 0.28; 'disk': 0.29; 'unicode': 0.29; 'lists': 0.29; 'consisting': 0.30; 'construct': 0.30; 'looks': 0.31; "can't": 0.32; 'does': 0.33; 'to:addr:python-list': 0.33; 'things': 0.33; 'chris': 0.34; 'characters': 0.34; 'there': 0.35; 'using': 0.35; 'translate': 0.36; 'probably': 0.36; 'uses': 0.36; 'received:google.com': 0.37; 'change': 0.37; 'received:209.85': 0.37; 'ways': 0.37; 'another': 0.37; 'but': 0.38; 'subject:: ': 0.38; 'should': 0.39; 'received:209': 0.39; 'system.': 0.39; 'to:addr:python.org': 0.39; 'really': 0.40; 'hope': 0.60; 'high': 0.67; 'ways,': 0.67; 'concept': 0.73; 'low': 0.73; 'stream': 0.77
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type:content-transfer-encoding; bh=G+ni121I+oqlak0XBxk+d5PgvJgzo9REwB0z9nceYso=; b=vRxa7jvdqbHRs0/Mgd06DT5wPrC3dvTkYhPsuE7Kk25dlWLU99X5wPqTTgiJfdZ2DZ iVUIhU42KPDYqdzoLge86EYEEYwsmJqhTBj5YlviwucWJkaqUMrpYSqQg20CbZT5lAT9 wQunJit5UEa37mDNd0OdeImkvSTtc1AyEexl0=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=OPAUsglUUPkqTkpSH0F4Rh00MxYa+pfc039JQMSOQf3CqGjEsw93/GbEAvN+vW/p6C PBp8ACVLgtzmFlr+RNkXoA9HziEwRtvTbzELejnis2UTO9DfT9sD/UVoF2xWYRYTssHH q9cC2RNQ0lWhG4Ie/i/Zo+5sogTXHufXJC6jw=
MIME-Version: 1.0
In-Reply-To: <0604E20B5F6F2F4784C9C8C71C5DD4DD2E33300F5D@EMARC112VS01.exchad.jpmchase.net>
References: <4de40ee8$0$6623$9b4e6d93@newsspool2.arcor-online.net> <mailman.2315.1306841548.9059.python-list@python.org> <4de50cfd$0$6538$9b4e6d93@newsspool4.arcor-online.net> <0604E20B5F6F2F4784C9C8C71C5DD4DD2E33300F5D@EMARC112VS01.exchad.jpmchase.net>
Date: Wed, 1 Jun 2011 03:19:52 +1000
Subject: Re: sqlalchemy and Unicode strings: errormessage
From: Chris Angelico <rosuav@gmail.com>
To: python-list@python.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.2326.1306862395.9059.python-list@python.org>
Lines: 30
NNTP-Posting-Host: 82.94.164.166
X-Trace: 1306862395 news.xs4all.nl 49178 [::ffff:82.94.164.166]:50466
X-Complaints-To: abuse@xs4all.nl
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:6746

On Wed, Jun 1, 2011 at 2:31 AM, Prasad, Ramit <ramit.prasad@jpmchase.com> w=
rote:
>>line =3D unicode(line.strip(),'utf8')
>>and now i get really utf8-strings. It does work but i dont know why it wo=
rks. For me it looks like i change an utf8-string to an utf8-string.
>
>
> I would like to point out that UTF-8 is not exactly "Unicode". From what =
I understand, Unicode is a standard while UTF-8 is like an implementation o=
f that standard (called an encoding). Being able to convert to Unicode (the=
 standard) should mean you are then able to convert to any encoding that su=
pports the Unicode characters used.

Unicode defines characters; UTF-8 is one way (of many) to represent
those characters in bytes. UTF-16 and UTF-32 are other ways of
representing those characters in bytes, and internally, Python
probably uses one of them - but there is no guarantee, and you should
never need to know. Unicode strings can be stored in memory and
manipulated in various ways, but they're a high level construct on par
with lists and dictionaries - they can't be stored on disk or
transmitted to another computer without using an encoding system.

UTF-8 is an efficient way to translate Unicode text consisting
primarily of low codepoint characters into bytes. It's not so much an
implementation of Unicode as a means of converting a mythical concept
of "Unicode characters" into a concrete stream of bytes.

Hope that clarifies things a little!

Chris Angelico