Path: csiph.com!news.mixmin.net!newsreader4.netcologne.de!news.netcologne.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: MRAB <python@mrabarnett.plus.com>
Newsgroups: comp.lang.python
Subject: Re: Capturing the bad codes that raise UnicodeError exceptions during decoding
Date: Thu, 4 Aug 2016 20:33:12 +0100
Lines: 51
Message-ID: <mailman.175.1470339200.6033.python-list@python.org>
References: <1470336426.893449.686225577.24458777@webmail.messagingengine.com> <CAPTjJmp-RCZB2J+GVdVNVM56Nhe-4y9GL9MWW+L-RnpNGziFfw@mail.gmail.com> <1470338568.900965.686261921.548DDA14@webmail.messagingengine.com> <fa6f6ec9-bf33-f891-959e-91cf945990b8@mrabarnett.plus.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0
In-Reply-To: <1470338568.900965.686261921.548DDA14@webmail.messagingengine.com>
Precedence: list
Xref: csiph.com comp.lang.python:112347

On 2016-08-04 20:22, Malcolm Greene wrote:
> Hi Chris,
>
> Thanks for your suggestions. I would like to capture the specific bad
> codes *before* they get replaced. So if a line of text has 10 bad codes
> (each one raising UnicodeError), I would like to track each exception's
> bad code but still return a valid decode line when finished.
>
> My goal is to count the total number of UnicodeExceptions within a file
> (as a data quality metric) and track the frequency of specific bad
> code's (via a collections.counter dict) to see if there's a pattern that
> can be traced to bad upstream process.
>
You could catch the UnicodeDecodeError exception and look at its attributes:

try:
     b'\x80'.decode('utf-8')
except UnicodeDecodeError as e:
     print('Failed to decode')
     print('e.start is', e.start)
     print('e.end is', e.end)
else:
     print('Decoded successfully')


It prints:

Failed to decode
e.start is 0
e.end is 1

> Malcolm
>
> <snipped>
> Remove them? Not sure what you mean, exactly; but would an
> errors="backslashreplace" decode do the job? Something like (assuming
> you use Python 3):
>
> def read_dirty_file(fn):
>     with open(fn, encoding="utf-8", errors="backslashreplace") as f:
>         for row in csv.DictReader(f):
>             process(row)
>
> You'll get Unicode text, but any bytes that don't make sense in UTF-8
> will be represented as eg \x80, with an actual backslash. Or use
> errors="replace" to hide them all behind U+FFFD, or other forms of
> error handling. That'll get done at a higher level than the CSV
> reader, like you suggest.
> </snipped>
>