Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: Peter Otten <__peter__@web.de>
Newsgroups: comp.lang.python
Subject: Re: getting fileinput to do errors='ignore' or 'replace'?
Date: Thu, 03 Dec 2015 17:11:37 +0100
Organization: None
Lines: 57
Message-ID: <mailman.174.1449159118.14615.python-list@python.org>
References: <fn26jcxltl.ln2@news.ducksburg.com> <8336jcxi2m.ln2@news.ducksburg.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7Bit
User-Agent: KNode/4.13.3
Precedence: list
Xref: csiph.com comp.lang.python:99966

Adam Funk wrote:

> On 2015-12-03, Adam Funk wrote:
> 
>> I'm having trouble with some input files that are almost all proper
>> UTF-8 but with a couple of troublesome characters mixed in, which I'd
>> like to ignore instead of throwing ValueError.  I've found the
>> openhook for the encoding
>>
>> for line in fileinput.input(options.files,
>> openhook=fileinput.hook_encoded("utf-8")):
>>     do_stuff(line)
>>
>> which the documentation describes as "a hook which opens each file
>> with codecs.open(), using the given encoding to read the file", but
>> I'd like codecs.open() to also have the errors='ignore' or
>> errors='replace' effect.  Is it possible to do this?
> 
> I forgot to mention: this is for Python 2.7.3 & 2.7.10 (on different
> machines).

Have a look at the source of fileinput.hook_encoded:

def hook_encoded(encoding):
    import io
    def openhook(filename, mode):
        mode = mode.replace('U', '').replace('b', '') or 'r'
        return io.open(filename, mode, encoding=encoding, newline='')
    return openhook

You can use it as a template to write your own factory function:

def my_hook_encoded(encoding, errors=None):
    import io
    def openhook(filename, mode):
        mode = mode.replace('U', '').replace('b', '') or 'r'
        return io.open(
            filename, mode, 
            encoding=encoding, newline='', 
            errors=errors)
    return openhook

for line in fileinput.input(
        options.files,
        openhook=my_hook_encoded("utf-8", errors="ignore")):
    do_stuff(line)

Another option is to create the function on the fly:

for line in fileinput.input(
        options.files,
        openhook=functools.partial(
            io.open, encoding="utf-8", errors="replace")):
    do_stuff(line)

(codecs.open() instead of io.open() should also work)