Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #98089 > unrolled thread

Unbuffered stderr in Python 3

Back to article view | Back to comp.lang.python

Contents

  Unbuffered stderr in Python 3 Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-11-02 18:52 +1100
    Re: Unbuffered stderr in Python 3 Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> - 2015-11-02 11:48 +0100
    Re: Unbuffered stderr in Python 3 Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> - 2015-11-02 11:54 +0100
    Re: Unbuffered stderr in Python 3 Nobody <nobody@nowhere.invalid> - 2015-11-03 07:10 +0000
      Re: Unbuffered stderr in Python 3 Random832 <random832@fastmail.com> - 2015-11-03 10:00 -0500
      Re: Unbuffered stderr in Python 3 Chris Angelico <rosuav@gmail.com> - 2015-11-04 02:42 +1100
      Re: Unbuffered stderr in Python 3 Terry Reedy <tjreedy@udel.edu> - 2015-11-03 17:25 -0500
        Re: Unbuffered stderr in Python 3 Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-11-04 14:18 +1100
          Re: Unbuffered stderr in Python 3 Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> - 2015-11-04 09:19 +0100
            Re: Unbuffered stderr in Python 3 Steven D'Aprano <steve@pearwood.info> - 2015-11-04 21:24 +1100
              Re: Unbuffered stderr in Python 3 Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> - 2015-11-04 11:43 +0100
              Re: Unbuffered stderr in Python 3 Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> - 2015-11-04 12:49 +0100
              Re: Unbuffered stderr in Python 3 Random832 <random832@fastmail.com> - 2015-11-04 08:43 -0500
          Re: Unbuffered stderr in Python 3 Terry Reedy <tjreedy@udel.edu> - 2015-11-04 18:00 -0500
    Re: Unbuffered stderr in Python 3 srinivas devaki <mr.eightnoteight@gmail.com> - 2015-11-06 19:10 +0530

#98089 — Unbuffered stderr in Python 3

In Python 2, stderr is unbuffered.

In most other environments (the shell, C...) stderr is unbuffered.

It is usually considered a bad, bad thing for stderr to be buffered. What 
happens if your application is killed before the buffer fills up? The errors 
in the buffer will be lost.

So how come Python 3 has line buffered stderr? And more importantly, how can 
I turn buffering off?

I don't want to use the -u unbuffered command line switch, because that 
effects stdout as well. I'm happy for stdout to remain buffered.


Here is the function I'm using to test this in in the interactive 
interpreter:

import sys, time

def test():
    # Simulate a slow calculation that prints status and/or error
    # messages to stderr.
    for i in range(10):
        print(i, file=sys.stderr, end="")
        time.sleep(2)
    print("", file=sys.stderr)


Running it pauses for 20 seconds, then displays "0123456789" in one go. What 
I expect is to see the digits arrive individually.

I tried this:

py> sys.stderr.line_buffering = False
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: readonly attribute


Next I tried this:

py> sys.stderr.buffer.line_buffering = False


which succeeded, but has no effect on print: it still buffers.




-- 
Steve

[toc] | [next] | [standalone]

#98093

On 02.11.2015 08:52, Steven D'Aprano wrote:
> In Python 2, stderr is unbuffered.
>
> In most other environments (the shell, C...) stderr is unbuffered.
>
> It is usually considered a bad, bad thing for stderr to be buffered. What
> happens if your application is killed before the buffer fills up? The errors
> in the buffer will be lost.
>
> So how come Python 3 has line buffered stderr? And more importantly, how can
> I turn buffering off?
>

I cannot comment on your first question, but unbuffered writing to 
sys.stderr is possible just like for any other buffered file object. You 
just call its flush method after writing.

Since Python3.3, the print function has a flush keyword argument that 
accepts a boolean and lets you do just this. Rewrite your example as:

import sys, time

def test():
# Simulate a slow calculation that prints status and/or error
# messages to stderr.
for i in range(10):
_print(i, file=sys.stderr, end="", flush=True)
time.sleep(2)
print("", file=sys.stderr)

and it should do what you want.


Before Python 3.3, if I needed an unbuffered print function, I used a 
wrapper to call flush for me, so instead of:

>
> import sys, time
>
> def test():
>      # Simulate a slow calculation that prints status and/or error
>      # messages to stderr.
>      for i in range(10):
>          print(i, file=sys.stderr, end="")
>          time.sleep(2)
>      print("", file=sys.stderr)
>

I'd use (and still do this if I need compatibility with 3.2):

import sys, time

def _print (*args, **kwargs):
    file = kwargs.get('file', sys.stdout)
    print(*args, **kwargs)
    file.flush()

def test():
     # Simulate a slow calculation that prints status and/or error
     # messages to stderr.
     for i in range(10):
         _print(i, file=sys.stderr, end="")
         time.sleep(2)
     print("", file=sys.stderr)

Best,
Wolfgang

[toc] | [prev] | [next] | [standalone]

#98094

On 02.11.2015 11:48, Wolfgang Maier wrote:
>
> Since Python3.3, the print function has a flush keyword argument that
> accepts a boolean and lets you do just this. Rewrite your example as:
>
> import sys, time
>
> def test():
> # Simulate a slow calculation that prints status and/or error
> # messages to stderr.
> for i in range(10):
> _print(i, file=sys.stderr, end="", flush=True)
> time.sleep(2)
> print("", file=sys.stderr)
>
> and it should do what you want.

sorry for this mess (copy pasted from the wrong source).
The code should be:

def test():
     # Simulate a slow calculation that prints status and/or error
     # messages to stderr.
     for i in range(10):
         print(i, file=sys.stderr, end="", flush=True)
         time.sleep(2)
     print("", file=sys.stderr)

[toc] | [prev] | [next] | [standalone]

#98135

On Mon, 02 Nov 2015 18:52:55 +1100, Steven D'Aprano wrote:

> In Python 2, stderr is unbuffered.
> 
> In most other environments (the shell, C...) stderr is unbuffered.
> 
> It is usually considered a bad, bad thing for stderr to be buffered. What
> happens if your application is killed before the buffer fills up? The
> errors in the buffer will be lost.
> 
> So how come Python 3 has line buffered stderr? And more importantly, how
> can I turn buffering off?

It's probably related to the fact that std{in,out,err} are Unicode
streams. 

	> type(sys.stderr)
	<class '_io.TextIOWrapper'>
	> type(sys.stderr.buffer)
	<class '_io.BufferedWriter'>
	> type(sys.stderr.buffer.raw)
	<class '_io.FileIO'>

It appears that you can turn it off with:

	sys.stderr = io.TextIOWrapper(sys.stderr.buffer.raw)
or:
	sys.stderr = io.TextIOWrapper(sys.stderr.detach().detach())

This results in a sys.stderr which appears to work and whose
.line_buffering property is False.

[toc] | [prev] | [next] | [standalone]

#98162

Nobody <nobody@nowhere.invalid> writes:

> It's probably related to the fact that std{in,out,err} are Unicode
> streams. 

There's no fundamental reason a Unicode stream should have to be line
buffered. If it's "related", it's only in that an oversight was made in
the course of making that change.

[toc] | [prev] | [next] | [standalone]

#98166

On Wed, Nov 4, 2015 at 2:00 AM, Random832 <random832@fastmail.com> wrote:
> Nobody <nobody@nowhere.invalid> writes:
>
>> It's probably related to the fact that std{in,out,err} are Unicode
>> streams.
>
> There's no fundamental reason a Unicode stream should have to be line
> buffered. If it's "related", it's only in that an oversight was made in
> the course of making that change.

Yep. Unicode *input* streams need to be buffered, but *output* can
always be insta-flushed. The only significance of Unicode to output is
that a single character may cause multiple bytes to be output; and
since output can block for even a single byte, it should be no
different.

+1 for making sys.stderr unbuffered.

ChrisA

[toc] | [prev] | [next] | [standalone]

#98191

On 11/3/2015 10:42 AM, Chris Angelico wrote:
> On Wed, Nov 4, 2015 at 2:00 AM, Random832 <random832@fastmail.com> wrote:
>> Nobody <nobody@nowhere.invalid> writes:
>>
>>> It's probably related to the fact that std{in,out,err} are Unicode
>>> streams.
>>
>> There's no fundamental reason a Unicode stream should have to be line
>> buffered. If it's "related", it's only in that an oversight was made in
>> the course of making that change.

The current behavior is not an 'oversight'.  I was considered, decided, 
and revisited in https://bugs.python.org/issue13601.  Guido: 
"Line-buffering should be good enough since in practice errors messages 
are always terminated by a newline."  If not, print(part_line, 
file=sys.stderr, flush=True) works for the unusual case.

> Yep. Unicode *input* streams need to be buffered, but *output* can
> always be insta-flushed. The only significance of Unicode to output is
> that a single character may cause multiple bytes to be output; and
> since output can block for even a single byte, it should be no
> different.
>
> +1 for making sys.stderr unbuffered.

-
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#98201

On Wednesday 04 November 2015 09:25, Terry Reedy wrote:

> On 11/3/2015 10:42 AM, Chris Angelico wrote:
>> On Wed, Nov 4, 2015 at 2:00 AM, Random832 <random832@fastmail.com> wrote:
>>> Nobody <nobody@nowhere.invalid> writes:
>>>
>>>> It's probably related to the fact that std{in,out,err} are Unicode
>>>> streams.
>>>
>>> There's no fundamental reason a Unicode stream should have to be line
>>> buffered. If it's "related", it's only in that an oversight was made in
>>> the course of making that change.
> 
> The current behavior is not an 'oversight'.  I was considered, decided,
> and revisited in https://bugs.python.org/issue13601.  Guido:
> "Line-buffering should be good enough since in practice errors messages
> are always terminated by a newline."  If not, print(part_line,
> file=sys.stderr, flush=True) works for the unusual case.

This is one of the offending line from our code base:

print('<4>Suspicious answer "{}"!'.format(answer), file=sys.stderr)

So that ought to be terminated by a newline. And yet, the stderr output 
doesn't show up until the program exits.




-- 
Steve

[toc] | [prev] | [next] | [standalone]

#98212

On 04.11.2015 04:18, Steven D'Aprano wrote:
> On Wednesday 04 November 2015 09:25, Terry Reedy wrote:
>
>> On 11/3/2015 10:42 AM, Chris Angelico wrote:
>>> On Wed, Nov 4, 2015 at 2:00 AM, Random832 <random832@fastmail.com> wrote:
>>>> Nobody <nobody@nowhere.invalid> writes:
>>>>
>>>>> It's probably related to the fact that std{in,out,err} are Unicode
>>>>> streams.
>>>>
>>>> There's no fundamental reason a Unicode stream should have to be line
>>>> buffered. If it's "related", it's only in that an oversight was made in
>>>> the course of making that change.
>>
>> The current behavior is not an 'oversight'.  I was considered, decided,
>> and revisited in https://bugs.python.org/issue13601.  Guido:
>> "Line-buffering should be good enough since in practice errors messages
>> are always terminated by a newline."  If not, print(part_line,
>> file=sys.stderr, flush=True) works for the unusual case.
>
> This is one of the offending line from our code base:
>
> print('<4>Suspicious answer "{}"!'.format(answer), file=sys.stderr)
>
> So that ought to be terminated by a newline. And yet, the stderr output
> doesn't show up until the program exits.
>

For me, that prints immediately. Where is your output going? Console, 
file, ...?

[toc] | [prev] | [next] | [standalone]

#98218

On Wed, 4 Nov 2015 07:19 pm, Wolfgang Maier wrote:

> On 04.11.2015 04:18, Steven D'Aprano wrote:
>> On Wednesday 04 November 2015 09:25, Terry Reedy wrote:
>>
>>> On 11/3/2015 10:42 AM, Chris Angelico wrote:
>>>> On Wed, Nov 4, 2015 at 2:00 AM, Random832 <random832@fastmail.com>
>>>> wrote:
>>>>> Nobody <nobody@nowhere.invalid> writes:
>>>>>
>>>>>> It's probably related to the fact that std{in,out,err} are Unicode
>>>>>> streams.
>>>>>
>>>>> There's no fundamental reason a Unicode stream should have to be line
>>>>> buffered. If it's "related", it's only in that an oversight was made
>>>>> in the course of making that change.
>>>
>>> The current behavior is not an 'oversight'.  I was considered, decided,
>>> and revisited in https://bugs.python.org/issue13601.  Guido:
>>> "Line-buffering should be good enough since in practice errors messages
>>> are always terminated by a newline."  If not, print(part_line,
>>> file=sys.stderr, flush=True) works for the unusual case.
>>
>> This is one of the offending line from our code base:
>>
>> print('<4>Suspicious answer "{}"!'.format(answer), file=sys.stderr)
>>
>> So that ought to be terminated by a newline. And yet, the stderr output
>> doesn't show up until the program exits.
>>
> 
> For me, that prints immediately. Where is your output going? Console,
> file, ...?

Going to stderr, like the whole thread is about :-)


The output does eventually show up written to the console, but only after
Python exits.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#98219

On 04.11.2015 11:24, Steven D'Aprano wrote:
> On Wed, 4 Nov 2015 07:19 pm, Wolfgang Maier wrote:
>
>> On 04.11.2015 04:18, Steven D'Aprano wrote:
>>>
>>> This is one of the offending line from our code base:
>>>
>>> print('<4>Suspicious answer "{}"!'.format(answer), file=sys.stderr)
>>>
>>> So that ought to be terminated by a newline. And yet, the stderr output
>>> doesn't show up until the program exits.
>>>
>>
>> For me, that prints immediately. Where is your output going? Console,
>> file, ...?
>
> Going to stderr, like the whole thread is about :-)
>

I see :), but seriously, what I meant was: is it going to the console 
directly or is your code base redirecting sys.stderr to another buffered 
object first?

>
> The output does eventually show up written to the console, but only after
> Python exits.
>
>
>

[toc] | [prev] | [next] | [standalone]

#98221

On 04.11.2015 11:43, Wolfgang Maier wrote:
> On 04.11.2015 11:24, Steven D'Aprano wrote:
>> On Wed, 4 Nov 2015 07:19 pm, Wolfgang Maier wrote:
>>
>>> On 04.11.2015 04:18, Steven D'Aprano wrote:
>>>>
>>>> This is one of the offending line from our code base:
>>>>
>>>> print('<4>Suspicious answer "{}"!'.format(answer), file=sys.stderr)
>>>>
>>>> So that ought to be terminated by a newline. And yet, the stderr output
>>>> doesn't show up until the program exits.
>>>>
>>>
>>> For me, that prints immediately. Where is your output going? Console,
>>> file, ...?
>>
>> Going to stderr, like the whole thread is about :-)
>>
>
> I see :), but seriously, what I meant was: is it going to the console
> directly or is your code base redirecting sys.stderr to another buffered
> object first?
>

Standard I/O streams are line-buffered only if interactive (i.e., 
connected to a console), but block-buffered otherwise.
What does sys.stderr.isatty() return?

[toc] | [prev] | [next] | [standalone]

#98227

Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> writes:
> Standard I/O streams are line-buffered only if interactive (i.e.,
> connected to a console), but block-buffered otherwise.

That's not appropriate for stderr, nor is it justified by the argument
that Terry Reedy cited earlier. I had assumed he was making an implicit
claim that stderr is _always_ line-buffered, _even if_ it is not
interactive.

> What does sys.stderr.isatty() return?

There are many situations in which a nominally interactive process might
not have a "tty" for stderr on MS Windows. In an Emacs M-x shell buffer,
in a cygwin terminal (for a non-cygwin python), etc.

[toc] | [prev] | [next] | [standalone]

#98250

On 11/3/2015 10:18 PM, Steven D'Aprano wrote:
> On Wednesday 04 November 2015 09:25, Terry Reedy wrote:
>
>> On 11/3/2015 10:42 AM, Chris Angelico wrote:
>>> On Wed, Nov 4, 2015 at 2:00 AM, Random832 <random832@fastmail.com> wrote:
>>>> Nobody <nobody@nowhere.invalid> writes:
>>>>
>>>>> It's probably related to the fact that std{in,out,err} are Unicode
>>>>> streams.
>>>>
>>>> There's no fundamental reason a Unicode stream should have to be line
>>>> buffered. If it's "related", it's only in that an oversight was made in
>>>> the course of making that change.
>>
>> The current behavior is not an 'oversight'.  I was considered, decided,
>> and revisited in https://bugs.python.org/issue13601.  Guido:
>> "Line-buffering should be good enough since in practice errors messages
>> are always terminated by a newline."  If not, print(part_line,
>> file=sys.stderr, flush=True) works for the unusual case.
>
> This is one of the offending line from our code base:
>
> print('<4>Suspicious answer "{}"!'.format(answer), file=sys.stderr)
>
> So that ought to be terminated by a newline.

Or include 'flush=True' if you really want that to be a partial line.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#98353

On Mon, Nov 2, 2015 at 1:22 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
>
> So how come Python 3 has line buffered stderr? And more importantly, how can
> I turn buffering off?
>
> I don't want to use the -u unbuffered command line switch, because that
> effects stdout as well. I'm happy for stdout to remain buffered.
>
you can simply turn buffering off for stderr by redefining the print
function or declaring a new print function like


from functools import partial
print = partial(print, flush=True)
# or
from functools import partial
import sys
printerr = partial(print, flush=True, file=sys.stderr)

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web