Path: csiph.com!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder2.enfer-du-nord.net!newsfeed.kamp.net!newsfeed.kamp.net!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Sat, 29 Jun 2013 10:02:56 -0400
From: Dave Angel <d@davea.name>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130510 Thunderbird/17.0.6
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: MeCab UTF-8 Decoding Problem
References: <f4fe97e3-5949-4c52-97d7-4995b8891efd@googlegroups.com>
In-Reply-To: <f4fe97e3-5949-4c52-97d7-4995b8891efd@googlegroups.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.3990.1372514590.3114.python-list@python.org>
Lines: 55
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:49421

On 06/29/2013 07:29 AM, fobos3@gmail.com wrote:
> Hi,

Using Python 2.7 on Linux, presumably?  It'd be better to be explicit.

>
> I am trying to use a program called MeCab, which does syntax analysis on Japanese text. The problem I am having is that it returns a byte string and if I try to print it, it prints question marks for almost all characters. However, if I try to use .decide, it throws an error. Here is my code:

What do the MeCab docs say the tagger.parse byte string represents? 
Maybe it's not text at all.  But surely it's not utf-8.

>
> #!/usr/bin/python
> # -*- coding:utf-8 -*-
>
> import MeCab
> tagger = MeCab.Tagger("-Owakati")
> text = 'MeCabで遊んでみよう！'
>
> result = tagger.parse(text)
> print result
>
> result = result.decode('utf-8')
> print result
>
> And here is the output:
>
> MeCab �� �� ��んで�� �� ��う！
>
> Traceback (most recent call last):
>    File "test.py", line 11, in <module>
>      result = result.decode('utf-8')
>    File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
>      return codecs.utf_8_decode(input, errors, True)
> UnicodeDecodeError: 'utf8' codec can't decode bytes in position 6-7: invalid continuation byte
>
>
> ------------------
> (program exited with code: 1)
> Press return to continue
>
> Also my terminal is able to display Japanese characters properly. For example print '日本語' works perfectly fine.

Are your terminal and your text editor using utf-8, or something else? 
Can you put your print statement in the source file above, and it'll 
also work fine?

Are you actually running it from the terminal, or some GUI?  I notice 
you get "(program exited with code: 1)" and "Press return to continue". 
   Neither of those is standard terminal fare on any OS I know of.



-- 
DaveA