Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #99966

Re: getting fileinput to do errors='ignore' or 'replace'?

Show all headers | View raw

Adam Funk wrote:

> On 2015-12-03, Adam Funk wrote:
> 
>> I'm having trouble with some input files that are almost all proper
>> UTF-8 but with a couple of troublesome characters mixed in, which I'd
>> like to ignore instead of throwing ValueError.  I've found the
>> openhook for the encoding
>>
>> for line in fileinput.input(options.files,
>> openhook=fileinput.hook_encoded("utf-8")):
>>     do_stuff(line)
>>
>> which the documentation describes as "a hook which opens each file
>> with codecs.open(), using the given encoding to read the file", but
>> I'd like codecs.open() to also have the errors='ignore' or
>> errors='replace' effect.  Is it possible to do this?
> 
> I forgot to mention: this is for Python 2.7.3 & 2.7.10 (on different
> machines).

Have a look at the source of fileinput.hook_encoded:

def hook_encoded(encoding):
    import io
    def openhook(filename, mode):
        mode = mode.replace('U', '').replace('b', '') or 'r'
        return io.open(filename, mode, encoding=encoding, newline='')
    return openhook

You can use it as a template to write your own factory function:

def my_hook_encoded(encoding, errors=None):
    import io
    def openhook(filename, mode):
        mode = mode.replace('U', '').replace('b', '') or 'r'
        return io.open(
            filename, mode, 
            encoding=encoding, newline='', 
            errors=errors)
    return openhook

for line in fileinput.input(
        options.files,
        openhook=my_hook_encoded("utf-8", errors="ignore")):
    do_stuff(line)

Another option is to create the function on the fly:

for line in fileinput.input(
        options.files,
        openhook=functools.partial(
            io.open, encoding="utf-8", errors="replace")):
    do_stuff(line)

(codecs.open() instead of io.open() should also work)

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

getting fileinput to do errors='ignore' or 'replace'? Adam Funk <a24061@ducksburg.com> - 2015-12-03 15:12 +0000
  Re: getting fileinput to do errors='ignore' or 'replace'? Adam Funk <a24061@ducksburg.com> - 2015-12-03 15:18 +0000
    Re: getting fileinput to do errors='ignore' or 'replace'? Peter Otten <__peter__@web.de> - 2015-12-03 17:11 +0100
      Re: getting fileinput to do errors='ignore' or 'replace'? Adam Funk <a24061@ducksburg.com> - 2015-12-03 19:17 +0000
    Re: getting fileinput to do errors='ignore' or 'replace'? Terry Reedy <tjreedy@udel.edu> - 2015-12-03 11:48 -0500
      Re: getting fileinput to do errors='ignore' or 'replace'? Adam Funk <a24061@ducksburg.com> - 2015-12-03 19:21 +0000
    Re: getting fileinput to do errors='ignore' or 'replace'? Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2015-12-03 22:26 +0000
    Re: getting fileinput to do errors='ignore' or 'replace'? Serhiy Storchaka <storchaka@gmail.com> - 2015-12-04 10:34 +0200
    Re: getting fileinput to do errors='ignore' or 'replace'? Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2015-12-04 09:00 +0000
      Re: getting fileinput to do errors='ignore' or 'replace'? Adam Funk <a24061@ducksburg.com> - 2015-12-07 14:46 +0000
  Re: getting fileinput to do errors='ignore' or 'replace'? MRAB <python@mrabarnett.plus.com> - 2015-12-03 16:12 +0000
  Re: getting fileinput to do errors='ignore' or 'replace'? Laura Creighton <lac@openend.se> - 2015-12-03 17:46 +0100
    Re: getting fileinput to do errors='ignore' or 'replace'? Adam Funk <a24061@ducksburg.com> - 2015-12-03 19:17 +0000
      Re: getting fileinput to do errors='ignore' or 'replace'? Laura Creighton <lac@openend.se> - 2015-12-03 21:40 +0100

csiph-web