Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.001 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'else:': 0.03; 'subject:file': 0.07; 'subject:two': 0.07; 'versions.': 0.07; '34,': 0.09; 'iterate': 0.09; 'lines.': 0.09; 'performs': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'python': 0.11; 'suggest': 0.14; 'wrote': 0.14; '"*"': 0.16; '3):': 0.16; 'benjamin': 0.16; 'caching': 0.16; 'descriptor': 0.16; 'descriptors': 0.16; 'garbage': 0.16; 'iterator': 0.16; 'itertools': 0.16; 'objects.': 0.16; 'programmers.': 0.16; 'received:80.91.229.3': 0.16; 'received:plane.gmane.org': 0.16; 'rgb(255,': 0.16; 'roy': 0.16; 'true:': 0.16; 'whitespace.': 0.16; 'helvetica;': 0.16; 'ignore': 0.16; 'wrote:': 0.18; 'basically': 0.19; 'cheap': 0.19; '>>>': 0.22; 'import': 0.22; 'print': 0.22; 'header:User-Agent:1': 0.23; 'simpler': 0.24; 'text,': 0.24; '---': 0.24; 'holds': 0.26; 'post': 0.26; 'gets': 0.27; 'header:X -Complaints-To:1': 0.27; 'idea': 0.28; '0);': 0.29; 'am,': 0.29; 'possibility': 0.29; 'unix': 0.29; 'medium;': 0.30; "i'm": 0.30; 'work.': 0.31; 'easier': 0.31; 'lines': 0.31; '255,': 0.31; 'assumes': 0.31; 'sep': 0.31; 'file': 0.32; 'probably': 0.32; 'agreed': 0.32; 'skip:- 30': 0.32; 'another': 0.32; 'text': 0.33; 'open': 0.33; 'url:python': 0.33; 'rgb(0,': 0.33; "i'd": 0.34; 'subject:with': 0.35; 'but': 0.35; 'there': 0.35; 'version': 0.36; 'really': 0.36; 'doing': 0.36; 'charset:us-ascii': 0.36; 'possible': 0.36; 'url:org': 0.36; 'application': 0.37; 'skip:- 20': 0.37; 'two': 0.37; 'auto;': 0.38; 'minimum': 0.38; 'problems': 0.38; 'url:library': 0.38; 'to:addr:python-list': 0.38; 'expect': 0.39; 'to:addr:python.org': 0.39; 'received:org': 0.40; 'how': 0.40; 'dave': 0.60; 'most': 0.60; 'break': 0.61; "you're": 0.61; 'costs': 0.63; 'such': 0.63; 'more': 0.64; 'needs,': 0.65; 'talking': 0.65; 'temporary': 0.65; 'life': 0.66; 'believe': 0.68; 'approaches': 0.68; 'smith': 0.68; 'arial,': 0.74; 'helvetica,': 0.74; 'inline': 0.74; 'sans-serif;': 0.78; 'around,': 0.84; 'email addr:panix.com': 0.84; 'hanging': 0.84; 'oscar': 0.84; 'subject:over': 0.84; '2013,': 0.91; 'angel': 0.91 X-Injected-Via-Gmane: http://gmane.org/ To: python-list@python.org From: Dave Angel Subject: Re: iterating over a file with two pointers Date: Wed, 18 Sep 2013 20:07:39 +0000 (UTC) References: <3018b3d4-f914-4c89-9f26-cd4b2af32e73@googlegroups.com> <52B7F7EA-C7C4-4DB6-A93C-25F4C058EB58@panix.com> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Gmane-NNTP-Posting-Host: 174.32.174.35 User-Agent: XPN/1.2.6 (Street Spirit ; Linux) X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 88 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1379534879 news.xs4all.nl 15935 [2001:888:2000:d::a6]:34709 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:54399 On 18/9/2013 10:36, Roy Smith wrote: >> Dave Angel wrote (and I agreed with): >>> I'd suggest you open the file twice, and get two file objects. Then you >>> can iterate over them independently. > > > On Sep 18, 2013, at 9:09 AM, Oscar Benjamin wrote: >> There's no need to use OS resources by opening the file twice or to >> screw up the IO caching with seek(). > > There's no reason NOT to use OS resources. That's what the OS is there for; to make life easier on application programmers. Opening a file twice costs almost nothing. File descriptors are almost as cheap as whitespace. > >> Peter's version holds just as many lines as is necessary in an >> internal Python buffer and performs the minimum possible >> amount of IO. > > I believe by "Peter's version", you're talking about: > >> from itertools import islice, tee >> >> with open("tmp.txt") as f: >> while True: >> for outer in f: >> print outer, >> if "*" in outer: >> f, g = tee(f) >> for inner in islice(g, 3): >> print " ", inner, >> break >> else: >> break > > > There's this note from http://docs.python.org/2.7/library/itertools.html#itertools.tee: > >> This itertool may require significant auxiliary storage (depending on how much temporary data needs to be stored). In general, if one iterator uses most or all of the data before another iterator starts, it is faster to use list() instead of tee(). > > > I have no idea how that interacts with the pattern above where you call tee() serially. You're basically doing > > with open("my_file") as f: > while True: > f, g = tee(f) > > Are all of those g's just hanging around, eating up memory, while waiting to be garbage collected? I have no idea. But I do know that no such problems exist with the two file descriptor versions. > > > > > > >> I would expect this to be more >> efficient as well as less error-prone on Windows. >> >> >> Oscar >> > > > --- > Roy Smith > roy@panix.com > > > > > >
Dave Angel <davea@davea.name> wrote (and I agreed with):
I'd suggest you open the file twice, and get two file objects.  Then you
can iterate over them independently.

On Sep 18, 2013, at 9:09 AM, Oscar Benjamin wrote:
There's no need to use OS resources by opening the file twice or to
screw up the IO caching with seek().

There's no reason NOT to use OS resources.  That's what the OS is there for; to make life easier on application programmers.  Opening a file twice costs almost nothing.  File descriptors are almost as cheap as whitespace.

Peter's version holds just as many lines as is necessary in an
internal Python buffer and performs the minimum possible
amount of IO.

I believe by "Peter's version", you're talking about:

from itertools import islice, tee 

with open("tmp.txt") as f: 
    while True: 
        for outer in f: 
            print outer, 
            if "*" in outer: 
                f, g = tee(f) 
                for inner in islice(g, 3): 
                    print "   ", inner, 
                break 
        else: 
            break 


There's this note from http://docs.python.org/2.7/library/itertools.html#itertools.tee:

This itertool may require significant auxiliary storage (depending on how much temporary data needs to be stored). In general, if one iterator uses most or all of the data before another iterator starts, it is faster to use list() instead of tee().

I have no idea how that interacts with the pattern above where you call tee() serially.  You're basically doing

with open("my_file") as f:
while True:
f, g = tee(f)

Are all of those g's just hanging around, eating up memory, while waiting to be garbage collected?  I have no idea.  But I do know that no such problems exist with the two file descriptor versions.






I would expect this to be more
efficient as well as less error-prone on Windows.


Oscar


>

---
Roy Smith


>
>
> And if you're willing to ignore the possibility that the text file has unix line endings, I'm willing to ignore the possibility that the text file has a huge number of lines. Everything is MUCH simpler if one assumes readlines() will work. Most of these other approaches are much more complex than the OP probably needs, if he ever gets around to actually describing his requirements. BTW, please post in text, all that html is really annoying. -- DaveA