Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #49432

Re: MeCab UTF-8 Decoding Problem

Show all headers | View raw

On 29/06/2013 12:29, fobos3@gmail.com wrote:
> Hi,
>
> I am trying to use a program called MeCab, which does syntax analysis on Japanese text. The problem I am having is that it returns a byte string and if I try to print it, it prints question marks for almost all characters. However, if I try to use .decide, it throws an error. Here is my code:
>
> #!/usr/bin/python
> # -*- coding:utf-8 -*-
>
> import MeCab
> tagger = MeCab.Tagger("-Owakati")

This is a bytestring. Are you sure it shouldn't be a Unicode string
instead, i.e. u'MeCabで遊んでみよう！'?

> text = 'MeCabで遊んでみよう！'
>
> result = tagger.parse(text)
> print result
>
> result = result.decode('utf-8')
> print result
>
> And here is the output:
>
> MeCab �� �� ��んで�� �� ��う！
>
> Traceback (most recent call last):
>    File "test.py", line 11, in <module>
>      result = result.decode('utf-8')
>    File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
>      return codecs.utf_8_decode(input, errors, True)
> UnicodeDecodeError: 'utf8' codec can't decode bytes in position 6-7: invalid continuation byte
>
>
> ------------------
> (program exited with code: 1)
> Press return to continue
>
> Also my terminal is able to display Japanese characters properly. For example print '日本語' works perfectly fine.
>
> Any ideas?
>

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

MeCab UTF-8 Decoding Problem fobos3@gmail.com - 2013-06-29 04:29 -0700
  Re: MeCab UTF-8 Decoding Problem Giorgos Tzampanakis <giorgos.tzampanakis@gmail.com> - 2013-06-29 13:46 +0000
  Re: MeCab UTF-8 Decoding Problem Dave Angel <d@davea.name> - 2013-06-29 10:02 -0400
  Re: MeCab UTF-8 Decoding Problem Terry Reedy <tjreedy@udel.edu> - 2013-06-29 11:32 -0400
  Re: MeCab UTF-8 Decoding Problem Terry Reedy <tjreedy@udel.edu> - 2013-06-29 11:55 -0400
  Re: MeCab UTF-8 Decoding Problem MRAB <python@mrabarnett.plus.com> - 2013-06-29 17:12 +0100
  Re: MeCab UTF-8 Decoding Problem Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-06-29 16:20 +0000

csiph-web