Path: csiph.com!usenet.pasdenom.info!gegeweb.org!news.ecp.fr!feeder3.cambriumusenet.nl!feed.tweaknews.nl!194.109.133.84.MISMATCH!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Terry Reedy <tjreedy@udel.edu>
Subject: Re: helping with unicode
Date: Mon, 02 Jul 2012 21:39:32 -0400
References: <56e3cafd-ec4f-4ae4-ad6c-685f2d991403@googlegroups.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20120614 Thunderbird/13.0.1
In-Reply-To: <56e3cafd-ec4f-4ae4-ad6c-685f2d991403@googlegroups.com>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.1727.1341279615.4697.python-list@python.org>
Lines: 72
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:24800

On 7/2/2012 8:49 PM, self.python wrote:
> it's a simple source view program.
>
> the codec of the target website is utf-8
> so I read it and print the decoded

which re-encodes before printing

> --------------------------------------------------------------
> #-*-coding:utf8-*-
> import urllib2
>
> rf=urllib2.urlopen(r"http://gall.dcinside.com/list.php?id=programming")
>
> print rf.read().decode('utf-8')
>
> raw_input()
> ---------------------------------------------------------------
>
> It works fine on python shell

Do you mean the Windows Command Prompt shell?
>
> but when I make the file "wrong.py" and run it,
> Error rises.
>
> ----------------------------------------------------------------
> Traceback (most recent call last):
>    File "C:wrong.py", line 8, in <module>
>      print rf.read().decode('utf-8')
> UnicodeEncodeError: 'cp949' codec can't encode character u'u1368' in position 5
> 5122: illegal multibyte sequence
> ---------------------------------------------------------------------
>
> cp949 is the basic codec of sys.stdout and cmd.exe
> but I have no idea why it doesn't works.

cp949 is a Euro-Korean multibyte encoding whose mapping is given at
http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP949.TXT
u1368 is not in the mapping. There is no reason the utf-8 site would 
restrict itself to the cp949 subset.

Perhap it prints in the interpreter because 2.x uses errors = 'replace' 
rather than 'strict' (as in 3.x).

Try print rf.read().decode('utf-8').encode('cp949', errors = 'replace')
Non-cp949 chars will print as '?'.

> printing without decode('utf-8') works fine on IDLE

because IDLE encodes to utf-8, and x.decode('utf-8').encode('utf-8') == x

> but on cmd, it print broken characters

Printing utf-8 encoded bytes as if cp949 encoded bytes is pretty hilariour
>
> the question may look silly:(
  but I want to know what is the problem



  or how to print the not broken strings.
>
> thanks for reading.
>


-- 
Terry Jan Reedy