Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed1.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <53fd6a48$0$11111$c3e8da3@news.astraweb.com>
References: <53fd6a48$0$11111$c3e8da3@news.astraweb.com>
Date: Wed, 27 Aug 2014 16:02:57 +1000
Subject: Re: Reading from sys.stdin reads the whole file in
From: Chris Angelico <rosuav@gmail.com>
Cc: "python-list@python.org" <python-list@python.org>
Content-Type: text/plain; charset=UTF-8
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.13488.1409119385.18130.python-list@python.org>
Lines: 52
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:77104

On Wed, Aug 27, 2014 at 3:19 PM, Steven D'Aprano <steve@pearwood.info> wrote:
> When I pipe one to the other, I expect each line to be printed as they
> arrive, but instead they all queue up and happen at once:

You're seeing two different problems here. One is the flushing of
stdout in out.py, as Marko mentioned, but it's easily proven that
that's not the whole issue. Compare "python out.py" and "python
out.py|cat" - the latter will demonstrate whether or not it's getting
flushed properly (the former, where stdout is a tty, will always flush
correctly).

But even with that sorted, iterating over stdin has issues in Python
2. Here's a tweaked version of your files (note that I cut the sleeps
to 2 seconds, but the effect is the same):

rosuav@sikorsky:~$ cat out.py
import time

print("Hello...",flush=True)
time.sleep(2)
print("World!",flush=True)
time.sleep(2)
print("Goodbye!",flush=True)
rosuav@sikorsky:~$ cat slurp.py
from __future__ import print_function
import sys
import time

for line in sys.stdin:
        print(time.ctime(), line)
rosuav@sikorsky:~$ python3 out.py|python slurp.py
Wed Aug 27 16:00:16 2014 Hello...

Wed Aug 27 16:00:16 2014 World!

Wed Aug 27 16:00:16 2014 Goodbye!

rosuav@sikorsky:~$ python3 out.py|python3 slurp.py
Wed Aug 27 16:00:19 2014 Hello...

Wed Aug 27 16:00:21 2014 World!

Wed Aug 27 16:00:23 2014 Goodbye!

rosuav@sikorsky:~$


With a Py2 consumer, there's still buffering happening. With a Py3
consumer, it works correctly. How to control the Py2 buffering,
though, I don't know.

ChrisA