Path: csiph.com!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail From: Laura Creighton Newsgroups: comp.lang.python Subject: Re: getting fileinput to do errors='ignore' or 'replace'? Date: Thu, 03 Dec 2015 17:46:39 +0100 Lines: 27 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Trace: news.uni-berlin.de s1Ui+DZAXDs79GeeyVDmvA8esbxce/FEjUlV6Zy+lTAg== Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.000 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; "subject:' ": 0.07; 'subject:getting': 0.07; 'cc:addr:python-list': 0.09; 'agrees': 0.09; 'effect.': 0.09; 'received:openend.se': 0.09; 'received:theraft.openend.se': 0.09; 'subject:ignore': 0.09; 'useful,': 0.13; 'ignore': 0.14; 'encoding': 0.15; 'skip:f 30': 0.15; 'thu,': 0.15; 'adam': 0.16; 'add,': 0.16; 'approval,': 0.16; 'cc:addr:lac': 0.16; 'cc:addr:openend.se': 0.16; 'from:addr:lac': 0.16; 'from:addr:openend.se': 0.16; 'from:name:laura creighton': 0.16; 'hacked': 0.16; 'message-id:@fido.openend.se': 0.16; 'received:fido': 0.16; 'received:fido.openend.se': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; 'laura': 0.18; 'input': 0.18; '2015': 0.20; 'cc:addr:python.org': 0.20; 'cc:2**1': 0.22; 'mixed': 0.22; 'dec': 0.23; 'url:bugs': 0.24; "i've": 0.25; 'received:se': 0.29; 'cc:no real name:2**1': 0.29; "i'd": 0.31; 'url:python': 0.33; 'file': 0.34; 'this?': 0.34; 'so,': 0.35; 'trouble': 0.35; 'skip:> 10': 0.35; 'problem.': 0.35; 'but': 0.36; 'should': 0.36; 'instead': 0.36; 'url:org': 0.36; 'possible': 0.36; 'subject:: ': 0.37; 'being': 0.37; 'charset:us- ascii': 0.37; 'files': 0.38; 'some': 0.40; 'easy': 0.60; 'future': 0.60; 'header:Message-Id:1': 0.61; '>with': 0.84; 'header:In- reply-to:1': 0.84 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=openend.se; s=default; t=1449161205; bh=lXS4cSmnjLMtMwQZTbVp5tr1bud0jUJ942tEu9jovTs=; h=To:cc:From:Subject:In-reply-to:References:Date:From; b=pu3x5zS+HuaWm7qkGYwRJ/a5yJw9MVhBSosulkP76q5JJhX3mWjdZNuiGNP0gNUjH foeWDAOebgOyHUudcXL03rzOLyZBabtOWU7vQV704bsqA9TP34TGQxW20+sJ3g0yOD THmuXc/4GNhWBx2J+bAjYvH9He1ugwFyxVR6NU6Y= In-reply-to: Comments: In-reply-to Adam Funk message dated "Thu, 03 Dec 2015 15:12:15 +0000." Content-ID: <17980.1449161199.1@fido> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.3.9 (theraft.openend.se [82.96.5.2]); Thu, 03 Dec 2015 17:46:45 +0100 (CET) X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.20+ Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Xref: csiph.com comp.lang.python:99972 In a message of Thu, 03 Dec 2015 15:12:15 +0000, Adam Funk writes: >I'm having trouble with some input files that are almost all proper >UTF-8 but with a couple of troublesome characters mixed in, which I'd >like to ignore instead of throwing ValueError. I've found the >openhook for the encoding > >for line in fileinput.input(options.files, openhook=3Dfileinput.hook_enco= ded("utf-8")): > do_stuff(line) > >which the documentation describes as "a hook which opens each file >with codecs.open(), using the given encoding to read the file", but >I'd like codecs.open() to also have the errors=3D'ignore' or >errors=3D'replace' effect. Is it possible to do this? > >Thanks. This should be both easy to add, and useful, and I happen to know that fileinput is being hacked on by Serhiy Storchaka right now, who agrees that this would be easy. So, with his approval, I stuck this into the tracker. http://bugs.python.org/issue25788 = Future Pythons may not have the problem. Laura