Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: Oscar Benjamin <oscar.j.benjamin@gmail.com>
Newsgroups: comp.lang.python
Subject: Re: How to read from a file to an arbitrary delimiter efficiently?
Date: Sun, 28 Feb 2016 20:28:29 +0000
Lines: 39
Message-ID: <mailman.24.1456691337.9760.python-list@python.org>
References: <56cea44e$0$11128$c3e8da3@news.astraweb.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
In-Reply-To: <56cea44e$0$11128$c3e8da3@news.astraweb.com>
Precedence: list
Xref: csiph.com comp.lang.python:103660

On 25 February 2016 at 06:50, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
>
> I have a need to read to an arbitrary delimiter, which might be any of a
> (small) set of characters. For the sake of the exercise, lets say it is
> either ! or ? (for example).
>
> I want to read from files reasonably efficiently. I don't mind if there is a
> little overhead, but my first attempt is 100 times slower than the built-in
> "read to the end of the line" method.

You can get something much faster using mmap and searching for a
single delimiter:

def readuntil(m, delim):
    start = m.tell()
    index = m.find(delim, start)
    if index == -1:
        return m.read()
    else:
        return m.read(index - start)

def readmmap(f):
    m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
    f.seek(0)
    while True:
        chunk = readuntil(m, b'!') # Note byte-string
        if not chunk:
            return
        # Do stuff with chunk
        pass

My timing makes that ~7x slower than iterating over the lines of the
file but still around 100x faster than reading individual characters.
I'm not sure how to generalise it to looking for multiple delimiters
without dropping back to reading individual characters though.

--
Oscar