Path: csiph.com!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: Serhiy Storchaka <storchaka@gmail.com>
Newsgroups: comp.lang.python
Subject: Re: getting fileinput to do errors='ignore' or 'replace'?
Date: Fri, 4 Dec 2015 10:34:42 +0200
Lines: 31
Message-ID: <mailman.191.1449218096.14615.python-list@python.org>
References: <fn26jcxltl.ln2@news.ducksburg.com> <8336jcxi2m.ln2@news.ducksburg.com> <n3prpd$pt4$1@ger.gmane.org> <CAHVvXxQCuHQdt3W5yUc02n_8UL=PqtQhp8ro63T_u4_d0bnpDw@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
X-Trace: news.uni-berlin.de sXu541WrLH64i9dnEI2YEwjnK4r2OkbKrro/QDzvm3rg==
Return-Path: <python-python-list@m.gmane.org>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.000
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'context': 0.05; "subject:' ": 0.07; 'subject:getting': 0.07; 'benjamin': 0.09; 'closed.': 0.09; 'files:': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'subject:ignore': 0.09; 'whichever': 0.09; '2.7': 0.13; '"terry': 0.16; 'consume': 0.16; 'iteration': 0.16; 'iterators': 0.16; 'iterators,': 0.16; 'received:80.91.229.3': 0.16; 'received:io': 0.16; 'received:plane.gmane.org': 0.16; 'received:psf.io': 0.16; 'reedy"': 0.16; 'statement.': 0.16; 'wrote:': 0.16; '2015': 0.20; 'dec': 0.23; 'header:In-Reply-To:1': 0.24; 'module': 0.25; 'header:User-Agent:1': 0.26; "doesn't": 0.26; 'header:X-Complaints-To:1': 0.26; 'function': 0.28; "skip:' 10": 0.28; 'fine': 0.28; 'convert': 0.29; 'statement': 0.32; 'wrap': 0.33; 'open': 0.33; 'correctly': 0.34; 'file': 0.34; 'skip:c 30': 0.35; 'replace': 0.35; '(and': 0.36; 'to:addr:python- list': 0.36; 'subject:: ': 0.37; 'received:org': 0.37; 'files': 0.38; 'end': 0.39; 'does': 0.39; 'easily': 0.39; 'to:addr:python.org': 0.40; 'your': 0.60; 'charset:windows-1252': 0.62; 'managers': 0.63; 'guaranteed': 0.67; 'oscar': 0.84
X-Injected-Via-Gmane: http://gmane.org/
X-Gmane-NNTP-Posting-Host: 193.202.118.162
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:38.0) Gecko/20100101 Thunderbird/38.4.0
In-Reply-To: <CAHVvXxQCuHQdt3W5yUc02n_8UL=PqtQhp8ro63T_u4_d0bnpDw@mail.gmail.com>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.20+
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Xref: csiph.com comp.lang.python:99997

On 04.12.15 00:26, Oscar Benjamin wrote:
> On 3 Dec 2015 16:50, "Terry Reedy" <tjreedy@udel.edu> wrote:
>> fileinput is an ancient module that predates iterators (and generators)
> and context managers. Since by 2.7 open files are both context managers and
> line iterators, you can easily write your own multi-file line iteration
> that does exactly what you want.  At minimum:
>>
>> for file in files:
>>      with codecs.open(file, errors='ignore') as f
>>      # did not look up signature,
>>          for line in f:
>>              do_stuff(line)
>
> The above is fine but...
>
>> To make this reusable, wrap in 'def filelines(files):' and replace
> 'do_stuff(line)' with 'yield line'.
>
> That doesn't work entirely correctly as you end up yielding from inside a
> with statement. If the user of your generator function doesn't fully
> consume the generator then whichever file is currently open is not
> guaranteed to be closed.

You can convert the generator to context manager and use it in the with 
statement to guarantee closing.

with contextlib.closing(filelines(files)) as f:
     for line in f:
         ...