Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!rt.uk.eu.org!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
Sender: joshua.landau.ws@gmail.com
In-Reply-To: <523a67c3$0$29988$c3e8da3$5496439d@news.astraweb.com>
References: <3018b3d4-f914-4c89-9f26-cd4b2af32e73@googlegroups.com> <523a67c3$0$29988$c3e8da3$5496439d@news.astraweb.com>
From: Joshua Landau <joshua@landau.ws>
Date: Thu, 19 Sep 2013 08:04:51 +0100
Subject: Re: iterating over a file with two pointers
To: python-list <python-list@python.org>
Content-Type: text/plain; charset=UTF-8
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.145.1379574349.18130.python-list@python.org>
Lines: 51
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:54417

Although "tee" is most certainly preferable because IO is far slower
than the small amounts of memory "tee" will use, you do have this
option:

    def iterate_file_lines(file):
        """
        Iterate over lines in a file, unlike normal
        iteration this allows seeking.
        """
        while True:
            line = thefile.readline()
            if not line:
                break

            yield line


    thefile = open("/tmp/thefile")
    thelines = iterate_file_lines(thefile)

    for line in thelines:
        print("Outer:", repr(line))

        if is_start(line):
            outer_position = thefile.tell()

            for line in thelines:
                print("Inner:", repr(line))

                if is_end(line):
                    break

            thefile.seek(outer_position)

It's simpler than having two files but probably not faster, "tee" will
almost definitely be way better a choice (unless the subsections can't
fit in memory) and it forfeits being able to change up the order of
these things.

If you want to change up the order to another defined order, you can
think about storing the subsections, but if you want to support
independent iteration you'll need to seek before every "readline"
which is a bit silly.

Basically, read it all into memory like Steven D'Aprano suggested. If
you really don't want to, use "tee". If you can't handle non-constant
memory usage (really? You're reading lines, man) I'd suggest my
method. If you can't handle the inflexibility there, use multiple
files.

There, is that enough choices?