Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Thu, 27 Jun 2013 12:28:59 -0400
From: Dave Angel <davea@davea.name>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130510 Thunderbird/17.0.6
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: Devnagari Unicode Conversion Issues
References: <c8ea987a-a493-4adc-a35d-11a82f1bd03a@googlegroups.com> <02ea5055-7617-4db1-a3b7-82d155c6954d@googlegroups.com>
In-Reply-To: <02ea5055-7617-4db1-a3b7-82d155c6954d@googlegroups.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.3935.1372350553.3114.python-list@python.org>
Lines: 44
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:49336

On 06/27/2013 11:39 AM, darpan6aya wrote:
> That worked out. I was trying to encode it the entire time.
> Now I realise how silly I am.
>
> Thanks MRAB. Once Again. :D
>

you're not silly, it's a complex question.  MRAB is good at guessing 
which part is messing you up.

However, when you're writing a real Python program with a real text 
editor, and when you're not using a newsgroup in between to mangle or 
unmangle things, you have a few things to match up to get it right.


The file is just a bunch of bytes.  Those bytes are being inserted in 
there by your editor, and interpreted by the compiler.  So if you have a 
non-ASCII character on your keyboard and you hit it, the editor will 
decode it (from Unicode to byte(s)) and put it in the file.  If you tell 
the editor to use utf-8, then you also want to tell the compiler to 
decode it using utf-8.

The most polite way to do that looks something like:
# -*- coding: <encoding-name> -*-
# -*- coding: <utf-8> -*-

http://docs.python.org/release/2.7.5/reference/lexical_analysis.html#encoding-declarations

Once you've got that straight, you don't need to explicitly decode byte 
strings.  You can just use
   u"This is my string"

with whatever characters you need.  As long as the declarations match, 
this should "just work."  If the data comes from a byte string other 
than a literal string, you might need the more verbose form.

Your original message was sent in Western (ISO 8859-1), and MRAB's 
response was in utf-8, and my mail program decoded the string the same 
way.  However, I don't know anything about Devnagari, so I can't say if 
it looked reasonable here.


-- 
DaveA