Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #81743 > unrolled thread

Re: [Python-Dev] Unicode decode exception

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [Python-Dev] Unicode decode exception Michael Torrie <torriem@gmail.com> - 2014-11-30 21:33 -0700
    Re: [Python-Dev] Unicode decode exception wxjmfauth@gmail.com - 2014-12-01 23:28 -0800

#81743 — Re: [Python-Dev] Unicode decode exception

On 11/30/2014 09:19 PM, balaji marisetti wrote:
> The default encoding is "UTF-8". It works if I do:
> 
> with open("filename", errors="ignore") as f:
>     ....
> 
> So I think Python2, by default, ignores all errors whereas Python3 doesn't

Do you mean that the file is supposed to be utf-8 but isn't?  Because if
it is, you need to tell open about it in Python3 to make sure that's the
decoding scheme it uses. Otherwise you will see this decode error.

Also I'm not sure you posted the full traceback. Usually a
UnicodeDecodeError says something about an invalid byte while using a
specific unicode decoding scheme (say, ASCII, which can't handle utf-8
bytes).

[toc] | [next] | [standalone]

#81794

>>> 
>>> # input
>>> source_ascii = b'\x61\x62\x63'
>>> source_greek = b'\xe1\xe2\xe3'
>>> source_cyrillic = b'\xd0\xd1\xd2'
>>> buffer = ''
>>> buffer  = buffer + source_ascii.decode('ascii')
>>> buffer  = buffer + source_greek.decode('cp1253')
>>> buffer  = buffer + source_cyrillic.decode('iso-8859-5')
>>> len(buffer)
9
>>> # output
>>> sys.stdout.sethostencoding('cp1252')
>>> print(buffer.encode(sys.stdout.encoding, 'replace').\
...     decode(sys.stdout.encoding))
abc??????
>>> sys.stdout.sethostencoding('mac-roman')
>>> print(buffer.encode(sys.stdout.encoding, 'replace').\
...     decode(sys.stdout.encoding))
abc??????
>>> sys.stdout.sethostencoding('ascii')
>>> print(buffer.encode(sys.stdout.encoding, 'replace').\
...     decode(sys.stdout.encoding))
abc??????
>>> sys.stdout.sethostencoding('cp1253')
>>> print(buffer.encode(sys.stdout.encoding, 'replace').\
...     decode(sys.stdout.encoding))
abcαβγ???
>>> sys.stdout.sethostencoding('iso-8859-5')
>>> print(buffer.encode(sys.stdout.encoding, 'replace').\
...     decode(sys.stdout.encoding))
abc???абв
>>> sys.stdout.sethostencoding('utf-32-be')
>>> print(buffer.encode(sys.stdout.encoding, 'replace').\
...     decode(sys.stdout.encoding))
abcαβγабв
>>> 
>>> # typical decode error
>>> source_greek.decode('ascii')
Traceback (most recent call last):
  File "<eta last command>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe1 in position 0: ordinal not in 
range(128)

From a *Py32* gui interactive interpreter mimicking
platforms.

jmf

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web