Re: getting fileinput to do errors='ignore' or 'replace'?

Path	csiph.com!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From	Serhiy Storchaka <storchaka@gmail.com>
Newsgroups	comp.lang.python
Subject	Re: getting fileinput to do errors='ignore' or 'replace'?
Date	Fri, 4 Dec 2015 10:34:42 +0200
Lines	31
Message-ID	<mailman.191.1449218096.14615.python-list@python.org> (permalink)
References	<fn26jcxltl.ln2@news.ducksburg.com> <8336jcxi2m.ln2@news.ducksburg.com> <n3prpd$pt4$1@ger.gmane.org> <CAHVvXxQCuHQdt3W5yUc02n_8UL=PqtQhp8ro63T_u4_d0bnpDw@mail.gmail.com>
Mime-Version	1.0
Content-Type	text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding	7bit
X-Trace	news.uni-berlin.de sXu541WrLH64i9dnEI2YEwjnK4r2OkbKrro/QDzvm3rg==
Return-Path	<python-python-list@m.gmane.org>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.000
X-Spam-Evidence	'H': 1.00; 'S': 0.00; 'context': 0.05; "subject:' ": 0.07; 'subject:getting': 0.07; 'benjamin': 0.09; 'closed.': 0.09; 'files:': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'subject:ignore': 0.09; 'whichever': 0.09; '2.7': 0.13; '"terry': 0.16; 'consume': 0.16; 'iteration': 0.16; 'iterators': 0.16; 'iterators,': 0.16; 'received:80.91.229.3': 0.16; 'received:io': 0.16; 'received:plane.gmane.org': 0.16; 'received:psf.io': 0.16; 'reedy"': 0.16; 'statement.': 0.16; 'wrote:': 0.16; '2015': 0.20; 'dec': 0.23; 'header:In-Reply-To:1': 0.24; 'module': 0.25; 'header:User-Agent:1': 0.26; "doesn't": 0.26; 'header:X-Complaints-To:1': 0.26; 'function': 0.28; "skip:' 10": 0.28; 'fine': 0.28; 'convert': 0.29; 'statement': 0.32; 'wrap': 0.33; 'open': 0.33; 'correctly': 0.34; 'file': 0.34; 'skip:c 30': 0.35; 'replace': 0.35; '(and': 0.36; 'to:addr:python- list': 0.36; 'subject:: ': 0.37; 'received:org': 0.37; 'files': 0.38; 'end': 0.39; 'does': 0.39; 'easily': 0.39; 'to:addr:python.org': 0.40; 'your': 0.60; 'charset:windows-1252': 0.62; 'managers': 0.63; 'guaranteed': 0.67; 'oscar': 0.84
X-Injected-Via-Gmane	http://gmane.org/
X-Gmane-NNTP-Posting-Host	193.202.118.162
User-Agent	Mozilla/5.0 (X11; Linux i686; rv:38.0) Gecko/20100101 Thunderbird/38.4.0
In-Reply-To	<CAHVvXxQCuHQdt3W5yUc02n_8UL=PqtQhp8ro63T_u4_d0bnpDw@mail.gmail.com>
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.20+
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Xref	csiph.com comp.lang.python:99997

Show key headers only | View raw

On 04.12.15 00:26, Oscar Benjamin wrote:
> On 3 Dec 2015 16:50, "Terry Reedy" <tjreedy@udel.edu> wrote:
>> fileinput is an ancient module that predates iterators (and generators)
> and context managers. Since by 2.7 open files are both context managers and
> line iterators, you can easily write your own multi-file line iteration
> that does exactly what you want.  At minimum:
>>
>> for file in files:
>>      with codecs.open(file, errors='ignore') as f
>>      # did not look up signature,
>>          for line in f:
>>              do_stuff(line)
>
> The above is fine but...
>
>> To make this reusable, wrap in 'def filelines(files):' and replace
> 'do_stuff(line)' with 'yield line'.
>
> That doesn't work entirely correctly as you end up yielding from inside a
> with statement. If the user of your generator function doesn't fully
> consume the generator then whichever file is currently open is not
> guaranteed to be closed.

You can convert the generator to context manager and use it in the with 
statement to guarantee closing.

with contextlib.closing(filelines(files)) as f:
     for line in f:
         ...

Thread

getting fileinput to do errors='ignore' or 'replace'? Adam Funk <a24061@ducksburg.com> - 2015-12-03 15:12 +0000
  Re: getting fileinput to do errors='ignore' or 'replace'? Adam Funk <a24061@ducksburg.com> - 2015-12-03 15:18 +0000
    Re: getting fileinput to do errors='ignore' or 'replace'? Peter Otten <__peter__@web.de> - 2015-12-03 17:11 +0100
      Re: getting fileinput to do errors='ignore' or 'replace'? Adam Funk <a24061@ducksburg.com> - 2015-12-03 19:17 +0000
    Re: getting fileinput to do errors='ignore' or 'replace'? Terry Reedy <tjreedy@udel.edu> - 2015-12-03 11:48 -0500
      Re: getting fileinput to do errors='ignore' or 'replace'? Adam Funk <a24061@ducksburg.com> - 2015-12-03 19:21 +0000
    Re: getting fileinput to do errors='ignore' or 'replace'? Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2015-12-03 22:26 +0000
    Re: getting fileinput to do errors='ignore' or 'replace'? Serhiy Storchaka <storchaka@gmail.com> - 2015-12-04 10:34 +0200
    Re: getting fileinput to do errors='ignore' or 'replace'? Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2015-12-04 09:00 +0000
      Re: getting fileinput to do errors='ignore' or 'replace'? Adam Funk <a24061@ducksburg.com> - 2015-12-07 14:46 +0000
  Re: getting fileinput to do errors='ignore' or 'replace'? MRAB <python@mrabarnett.plus.com> - 2015-12-03 16:12 +0000
  Re: getting fileinput to do errors='ignore' or 'replace'? Laura Creighton <lac@openend.se> - 2015-12-03 17:46 +0100
    Re: getting fileinput to do errors='ignore' or 'replace'? Adam Funk <a24061@ducksburg.com> - 2015-12-03 19:17 +0000
      Re: getting fileinput to do errors='ignore' or 'replace'? Laura Creighton <lac@openend.se> - 2015-12-03 21:40 +0100

csiph-web