Path: csiph.com!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder2.enfer-du-nord.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed1.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Date: Mon, 10 Jun 2013 15:46:24 -0400
From: Dave Angel <davea@davea.name>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130510 Thunderbird/17.0.6
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: Popen and reading stdout in windows
References: <fc7fe12873ac476280f4d02714470fb5@exch.activenetwerx.com>
In-Reply-To: <fc7fe12873ac476280f4d02714470fb5@exch.activenetwerx.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.2982.1370893598.3114.python-list@python.org>
Lines: 45
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:47593

On 06/10/2013 02:37 PM, Joseph L. Casale wrote:
> I have a use where writing an interim file is not convenient and I was hoping to
> iterate through maybe 100k lines of output by a process as its generated or
> roughly anyways.
>
> Seems to be a common question on ST, and more easily solved in Linux.
> Anyone currently doing this with Python 2.7 in windows and can share some
> guidance?
>

You leave out an awful amount of detail.  I have no idea what ST is, so 
I'll have to guess your real problem.

You've got a process (myprog.exe) which generates a medium amount of 
output to stdout, and you want to process that data in a python program 
as it's being output, but without first writing it to a file.

If by process you meant grep, the answer would be as simple as

myprocess | grep  parm1 parm2

But you want to write something (not called grep) in Python.

myprocess | python myfilter.py

The question is how to write myfilter.py


Answer is to use stdin as you would a file.  it's already open for you, 
and it'll get the data as it's being generated (plus or minus some 
buffering).  So you can simply do something like:



import sys
for index, line in enumerate(sys.stdin):
     print index, line


This trivial "filter" adds a line number in front of every line.  But 
you could do anything there.  And you might not need enumerate if you 
don't care about the line number.

-- 
DaveA