Path: csiph.com!news.mixmin.net!npeer.de.kpn-eurorings.net!npeer-ng0.de.kpn-eurorings.net!fu-berlin.de!uni-berlin.de!not-for-mail
From: Tim Delaney <timothy.c.delaney@gmail.com>
Newsgroups: comp.lang.python
Subject: Re: How to read from a file to an arbitrary delimiter efficiently?
Date: Mon, 29 Feb 2016 08:00:22 +1100
Lines: 36
Message-ID: <mailman.26.1456693224.9760.python-list@python.org>
References: <56cea44e$0$11128$c3e8da3@news.astraweb.com> <CAHVvXxQ-iTv=OAWDtnzQ0BKG2mcSK7-bDKXiaVP2r1kSCm8+nA@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
In-Reply-To: <CAHVvXxQ-iTv=OAWDtnzQ0BKG2mcSK7-bDKXiaVP2r1kSCm8+nA@mail.gmail.com>
Precedence: list
Xref: csiph.com comp.lang.python:103662

On 29 February 2016 at 07:28, Oscar Benjamin <oscar.j.benjamin@gmail.com>
wrote:

> On 25 February 2016 at 06:50, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
> >
> > I have a need to read to an arbitrary delimiter, which might be any of a
> > (small) set of characters. For the sake of the exercise, lets say it is
> > either ! or ? (for example).
> >
> > I want to read from files reasonably efficiently. I don't mind if there
> is a
> > little overhead, but my first attempt is 100 times slower than the
> built-in
> > "read to the end of the line" method.
>
> You can get something much faster using mmap and searching for a
> single delimiter:
>
> My timing makes that ~7x slower than iterating over the lines of the
> file but still around 100x faster than reading individual characters.
> I'm not sure how to generalise it to looking for multiple delimiters
> without dropping back to reading individual characters though.
>

You can use an mmapped file as the input for regular expressions. May or
may not be particularly efficient.

Otherwise, if reading from a file I think read a chunk, and seek() back to
the delimiter is probably going to be most efficient whilst leaving the
file position just after the delimiter.

If reading from a stream, I think Chris' read a chunk and maintain an
internal buffer, and don't give access to the underlying stream.

Tim Delaney