Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #99987

Re: getting fileinput to do errors='ignore' or 'replace'?

Show key headers only | View raw

On 3 Dec 2015 16:50, "Terry Reedy" <tjreedy@udel.edu> wrote:
>
> On 12/3/2015 10:18 AM, Adam Funk wrote:
>>
>> On 2015-12-03, Adam Funk wrote:
>>
>>> I'm having trouble with some input files that are almost all proper
>>> UTF-8 but with a couple of troublesome characters mixed in, which I'd
>>> like to ignore instead of throwing ValueError.  I've found the
>>> openhook for the encoding
>>>
>>> for line in fileinput.input(options.files,
openhook=fileinput.hook_encoded("utf-8")):
>>>      do_stuff(line)
>>>
>>> which the documentation describes as "a hook which opens each file
>>> with codecs.open(), using the given encoding to read the file", but
>>> I'd like codecs.open() to also have the errors='ignore' or
>>> errors='replace' effect.  Is it possible to do this?
>>
>>
>> I forgot to mention: this is for Python 2.7.3 & 2.7.10 (on different
>> machines).
>
>
> fileinput is an ancient module that predates iterators (and generators)
and context managers. Since by 2.7 open files are both context managers and
line iterators, you can easily write your own multi-file line iteration
that does exactly what you want.  At minimum:
>
> for file in files:
>     with codecs.open(file, errors='ignore') as f
>     # did not look up signature,
>         for line in f:
>             do_stuff(line)

The above is fine but...

> To make this reusable, wrap in 'def filelines(files):' and replace
'do_stuff(line)' with 'yield line'.

That doesn't work entirely correctly as you end up yielding from inside a
with statement. If the user of your generator function doesn't fully
consume the generator then whichever file is currently open is not
guaranteed to be closed.

--
Oscar

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

getting fileinput to do errors='ignore' or 'replace'? Adam Funk <a24061@ducksburg.com> - 2015-12-03 15:12 +0000
  Re: getting fileinput to do errors='ignore' or 'replace'? Adam Funk <a24061@ducksburg.com> - 2015-12-03 15:18 +0000
    Re: getting fileinput to do errors='ignore' or 'replace'? Peter Otten <__peter__@web.de> - 2015-12-03 17:11 +0100
      Re: getting fileinput to do errors='ignore' or 'replace'? Adam Funk <a24061@ducksburg.com> - 2015-12-03 19:17 +0000
    Re: getting fileinput to do errors='ignore' or 'replace'? Terry Reedy <tjreedy@udel.edu> - 2015-12-03 11:48 -0500
      Re: getting fileinput to do errors='ignore' or 'replace'? Adam Funk <a24061@ducksburg.com> - 2015-12-03 19:21 +0000
    Re: getting fileinput to do errors='ignore' or 'replace'? Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2015-12-03 22:26 +0000
    Re: getting fileinput to do errors='ignore' or 'replace'? Serhiy Storchaka <storchaka@gmail.com> - 2015-12-04 10:34 +0200
    Re: getting fileinput to do errors='ignore' or 'replace'? Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2015-12-04 09:00 +0000
      Re: getting fileinput to do errors='ignore' or 'replace'? Adam Funk <a24061@ducksburg.com> - 2015-12-07 14:46 +0000
  Re: getting fileinput to do errors='ignore' or 'replace'? MRAB <python@mrabarnett.plus.com> - 2015-12-03 16:12 +0000
  Re: getting fileinput to do errors='ignore' or 'replace'? Laura Creighton <lac@openend.se> - 2015-12-03 17:46 +0100
    Re: getting fileinput to do errors='ignore' or 'replace'? Adam Funk <a24061@ducksburg.com> - 2015-12-03 19:17 +0000
      Re: getting fileinput to do errors='ignore' or 'replace'? Laura Creighton <lac@openend.se> - 2015-12-03 21:40 +0100

csiph-web