Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Mon, 07 Nov 2011 09:33:33 -0500
From: Dave Angel <d@davea.name>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Thunderbird/3.1.15
MIME-Version: 1.0
To: Jaroslav Dobrek <jaroslav.dobrek@gmail.com>
Subject: Re: read from file with mixed encodings in Python3
References: <406195f9-d51f-416a-b32c-e9ab07e219c7@n13g2000vbv.googlegroups.com>
In-Reply-To: <406195f9-d51f-416a-b32c-e9ab07e219c7@n13g2000vbv.googlegroups.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: python-list@python.org
Precedence: list
Reply-To: d@davea.name
Newsgroups: comp.lang.python
Message-ID: <mailman.2496.1320676445.27778.python-list@python.org>
Lines: 54
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:15414

On 11/07/2011 09:23 AM, Jaroslav Dobrek wrote:
> Hello,
>
> in Python3, I often have this problem: I want to do something with
> every line of a file. Like Python3, I presuppose that every line is
> encoded in utf-8. If this isn't the case, I would like Python3 to do
> something specific (like skipping the line, writing the line to
> standard error, ...)
>
> Like so:
>
> try:
>     ....
> except UnicodeDecodeError:
>    ...
>
> Yet, there is no place for this construction. If I simply do:
>
> for line in f:
>      print(line)
>
> this will result in a UnicodeDecodeError if some line is not utf-8,
> but I can't tell Python3 to stop:
>
> This will not work:
>
> for line in f:
>      try:
>          print(line)
>      except UnicodeDecodeError:
>          ...
>
> because the UnicodeDecodeError is caused in the "for line in f"-part.
>
> How can I catch such exceptions?
>
> Note that recoding the file before opening it is not an option,
> because often files contain many different strings in many different
> encodings.
>
> Jaroslav
A file with mixed encodings isn't a text file.  So open it with 'rb' 
mode, and use read() on it.  Find your own line-endings, since a given 
'\n' byte may or may not be a line-ending.

Once you've got something that looks like a line, explicitly decode it 
using utf-8.  Some invalid lines will give an exception and some will 
not.  But perhaps you've got some other gimmick to tell the encoding for 
each line.

-- 

DaveA