Capturing the bad codes that raise UnicodeError exceptions during decoding

Path	csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From	Malcolm Greene <python@bdurham.com>
Newsgroups	comp.lang.python
Subject	Capturing the bad codes that raise UnicodeError exceptions during decoding
Date	Thu, 04 Aug 2016 14:47:06 -0400
Lines	10
Message-ID	<mailman.170.1470336428.6033.python-list@python.org> (permalink)
References	<1470336426.893449.686225577.24458777@webmail.messagingengine.com>
Mime-Version	1.0
Content-Type	text/plain
Content-Transfer-Encoding	7bit
X-Trace	news.uni-berlin.de oXYzcXnvSdo1cr01o8eAAgZwX53Aq6A03pZdzPc9/WPQ==
Return-Path	<python@bdurham.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.009
X-Spam-Evidence	'H': 0.98; 'S': 0.00; 'csv': 0.09; 'received:internal': 0.09; 'subject: \n ': 0.15; 'decoding': 0.16; 'from:addr:python': 0.16; 'malcolm': 0.16; 'message- id:@webmail.messagingengine.com': 0.16; 'received:10.202': 0.16; 'received:10.202.2': 0.16; 'received:10.202.2.212': 0.16; 'received:66.111': 0.16; 'received:66.111.4': 0.16; 'received:io': 0.16; 'received:messagingengine.com': 0.16; 'received:psf.io': 0.16; 'passes': 0.18; 'pass': 0.22; 'seems': 0.23; 'appreciated.': 0.27; 'figure': 0.27; 'errors.': 0.27; 'subject:that': 0.29; "i'm": 0.30; 'raising': 0.33; 'wrap': 0.33; 'level': 0.35; 'to:addr:python-list': 0.36; 'received:10': 0.37; 'received:66': 0.38; 'thank': 0.38; 'files': 0.38; 'subject:the': 0.39; 'to:addr:python.org': 0.40; 'high': 0.60; 'skip:u 10': 0.61; 'header:Message-Id:1': 0.61; 'repeat': 0.67; 'dirty': 0.93
DKIM-Signature	v=1; a=rsa-sha1; c=relaxed/relaxed; d=bdurham.com; h= content-transfer-encoding:content-type:date:from:message-id :mime-version:subject:to:x-sasl-enc:x-sasl-enc; s=mesmtp; bh=1R0 T9rxEkuEWXePBoa4AGuRffW4=; b=YYJK/DuJzhxdMsVngH/MDkrdrhEPB3EuLTw dFC38H96McP21dbFkAEMItg7seO7UWFhbq9tCwG2awLS97DtLdSbuxl2znSrvymY Yz7ekXaxIkWO4WLto4bomg7Cg4X7pbn7Azyvfd12ID49Wtkou3lp2GRi4OcPX2qC fxxFBd3c=
DKIM-Signature	v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-sasl-enc :x-sasl-enc; s=smtpout; bh=1R0T9rxEkuEWXePBoa4AGuRffW4=; b=ucVZR HMCE2cl5s+PewAoNPqpUTDTi1jfTnvpepzuzDSfXOt9mu/mCZ8qtl9h4N2CPm83R sy0bcJd+AECJg24QGg36FjS78h003d/cfbzDcFc81aAQdkwPBSu1V5Mc0YxGkRsL iPxCHSLL6E/sCZyy4C5chwM9CyzgsP8mfOWq1c=
X-Sasl-Enc	b4bFeLIMrhpMOThPb09FieW/ZuHW6WbAgxDAVn7o3xyP 1470336426
X-Mailer	MessagingEngine.com Webmail Interface - ajax-b9085e99
X-Content-Filtered-By	Mailman/MimeDel 2.1.22
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.22
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID	<1470336426.893449.686225577.24458777@webmail.messagingengine.com>
Xref	csiph.com comp.lang.python:112342

Show key headers only | View raw

I'm processing a lot of dirty CSV files and would like to track the bad
codes that are raising UnicodeErrors. I'm struggling how to figure out
what the exact codes are so I can track them, them remove them, and then
repeat the decoding process for the current line until the line has been
fully decoded so I can pass this line on to the CSV reader. At a high
level it seems that I need to wrap the decoding of a line until it
passes with out any errors. Any suggestions appreciated.

Thank you,
Malcolm

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Capturing the bad codes that raise UnicodeError exceptions during decoding Malcolm Greene <python@bdurham.com> - 2016-08-04 14:47 -0400

csiph-web