Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!newsfeed.datemas.de!feeder.erje.net!eu.feeder.erje.net!xlned.com!feeder5.xlned.com!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Terry Reedy <tjreedy@udel.edu>
Subject: Re: MeCab UTF-8 Decoding Problem
Date: Sat, 29 Jun 2013 11:32:48 -0400
References: <f4fe97e3-5949-4c52-97d7-4995b8891efd@googlegroups.com> <51CEE910.4090701@davea.name>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130620 Thunderbird/17.0.7
In-Reply-To: <51CEE910.4090701@davea.name>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.3994.1372519986.3114.python-list@python.org>
Lines: 84
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:49427

On 6/29/2013 10:02 AM, Dave Angel wrote:
> On 06/29/2013 07:29 AM, fobos3@gmail.com wrote:
>> Hi,
>
> Using Python 2.7 on Linux, presumably?  It'd be better to be explicit.
>
>>
>> I am trying to use a program called MeCab, which does syntax analysis
>> on Japanese text.

It is generally nice to give a link when asking about 3rd party=20
software.  https://code.google.com/p/mecab/
In this case, nearly all the non-boilerplate text is Japanese ;-(.

 >> The problem I am having is that it returns a byte string

and the problem with bytes is that they can have any encoding.
In Python 2 (indicated by your print *statements*), a byte string is=20
just a string.

>> and if I try to print it, it prints question marks for almost
>> all characters. However, if I try to use .decide, it throws an error.
>> Here is my code:
>
> What do the MeCab docs say the tagger.parse byte string represents?
> Maybe it's not text at all.  But surely it's not utf-8.

https://mecab.googlecode.com/svn/trunk/mecab/doc/index.html
MeCab: Yet Another Part-of-Speech and Morphological Analyzer
followed by Japanese.

>> #!/usr/bin/python
>> # -*- coding:utf-8 -*-
>>
>> import MeCab
>> tagger =3D MeCab.Tagger("-Owakati")
>> text =3D 'MeCab=E3=81=A7=E9=81=8A=E3=82=93=E3=81=A7=E3=81=BF=E3=82=88=E3=
=81=86=EF=BC=81'

Parts of this appear in the output, as indicated by spaces.
'MeCab=E3=81=A7=E9=81=8A =E3=82=93=E3=81=A7 =E3=81=BF=E3=82=88 =E3=81=86=EF=
=BC=81'

>> result =3D tagger.parse(text)
>> print result
>>
>> result =3D result.decode('utf-8')
>> print result
>>
>> And here is the output:
>>
>> MeCab =EF=BF=BD=EF=BF=BD =EF=BF=BD=EF=BF=BD =EF=BF=BD=EF=BF=BD=E3=82=93=
=E3=81=A7=EF=BF=BD=EF=BF=BD =EF=BF=BD=EF=BF=BD =EF=BF=BD=EF=BF=BD=E3=81=86=
=EF=BC=81

Python normally prints bytes with ascii chars representing either=20
themselves or other values with hex escapes. This looks more like=20
unicode sent to a terminal with a limited character set. I would add

print type(result)

to be sure.

>> Traceback (most recent call last):
>>    File "test.py", line 11, in <module>
>>      result =3D result.decode('utf-8')
>>    File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
>>      return codecs.utf_8_decode(input, errors, True)
>> UnicodeDecodeError: 'utf8' codec can't decode bytes in position 6-7:
>> invalid continuation byte
>>
>>
>> ------------------
>> (program exited with code: 1)
>> Press return to continue
>>
>> Also my terminal is able to display Japanese characters properly. For
>> example print '=E6=97=A5=E6=9C=AC=E8=AA=9E' works perfectly fine.


--=20
Terry Jan Reedy