Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Peter Otten <__peter__@web.de>
Subject: Re: read from file with mixed encodings in Python3
Date: Mon, 07 Nov 2011 15:42:47 +0100
Organization: None
References: <406195f9-d51f-416a-b32c-e9ab07e219c7@n13g2000vbv.googlegroups.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8Bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.2497.1320676977.27778.python-list@python.org>
Lines: 58
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:15415

Jaroslav Dobrek wrote:

> Hello,
> 
> in Python3, I often have this problem: I want to do something with
> every line of a file. Like Python3, I presuppose that every line is
> encoded in utf-8. If this isn't the case, I would like Python3 to do
> something specific (like skipping the line, writing the line to
> standard error, ...)
> 
> Like so:
> 
> try:
>    ....
> except UnicodeDecodeError:
>   ...
> 
> Yet, there is no place for this construction. If I simply do:
> 
> for line in f:
>     print(line)
> 
> this will result in a UnicodeDecodeError if some line is not utf-8,
> but I can't tell Python3 to stop:
> 
> This will not work:
> 
> for line in f:
>     try:
>         print(line)
>     except UnicodeDecodeError:
>         ...
> 
> because the UnicodeDecodeError is caused in the "for line in f"-part.
> 
> How can I catch such exceptions?
> 
> Note that recoding the file before opening it is not an option,
> because often files contain many different strings in many different
> encodings.

I don't see those files often, but I think they are all seriously broken. 
There's no way to recover the information from files with unknown mixed 
encodings. However, here's an approach that may sometimes work: 

>>> with open("tmp.txt", "rb") as f:
...     for line in f:
...             try:
...                     line = "UTF-8 " + line.decode("utf-8")
...             except UnicodeDecodeError:
...                     line = "Latin-1 " + line.decode("latin-1")
...             print(line, end="")
...
UTF-8 äöü
Latin-1 äöü
UTF-8 äöü