Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #79254 > unrolled thread

Flush stdin

Back to article view | Back to comp.lang.python

Contents

  Flush stdin Empty Account <emptya45@gmail.com> - 2014-10-17 12:38 +0100
    Re: Flush stdin Nobody <nobody@nowhere.invalid> - 2014-10-18 17:55 +0100
      Re: Flush stdin Tim Chase <python.list@tim.thechases.com> - 2014-10-18 12:32 -0500
        Re: Flush stdin Nobody <nobody@nowhere.invalid> - 2014-10-19 02:11 +0100
          Re: Flush stdin Dan Stromberg <drsalists@gmail.com> - 2014-10-18 18:34 -0700
          Re: Flush stdin Dan Stromberg <drsalists@gmail.com> - 2014-10-18 18:42 -0700
            Re: Flush stdin Nobody <nobody@nowhere.invalid> - 2014-10-22 03:49 +0100
              Re: Flush stdin Dan Stromberg <drsalists@gmail.com> - 2014-10-21 20:11 -0700
      Re: Flush stdin MRAB <python@mrabarnett.plus.com> - 2014-10-18 18:55 +0100
      Re: Flush stdin Cameron Simpson <cs@zip.com.au> - 2014-10-19 08:01 +1100
      Re: Flush stdin Terry Reedy <tjreedy@udel.edu> - 2014-10-18 18:35 -0400

#79254 — Flush stdin

[Multipart message — attachments visible in raw view] — view raw

Hi,

I am using netcat to listen to a port and python to read stdin and print to
the console.

nc -l 2003 | python print_metrics.py

sys.stdin.flush() doesn’t seem to flush stdin, so I am using the termios
module.

while True:
   input = sys.stdin.readline()
   # do some parsing
   …
   sys.stdout.write(parsed_data)
   time.sleep(3)
   termios.tcflush(sys.stdin, termios.TCIOFLUSH)


I am receiving this exception
termios.error: (25, 'Inappropriate ioctl for device')


I will be using this script on Unix based systems and I wondered what
approach I could use
to flush stdin?

Many Thanks

Aidy

[toc] | [next] | [standalone]

#79272

On Fri, 17 Oct 2014 12:38:54 +0100, Empty Account wrote:

> I am using netcat to listen to a port and python to read stdin and print
> to the console.
> 
> nc -l 2003 | python print_metrics.py
> 
> sys.stdin.flush() doesn’t seem to flush stdin,

You can't "flush" an input stream.

> so I am using the termios module.

> I am receiving this exception
> termios.error: (25, 'Inappropriate ioctl for device')

termios only works on terminals, not pipes.

It's a safe bet that your problem is that "nc" isn't flushing its stdout
after each line (this is the default behaviour for stdio streams which
don't correspond to a terminal).

Check whether "nc" has a flag to line-buffer its output. If it doesn't,
the simplest solution is probably to write a Python script which creates a
pseudo-tty (using the "pty" module) and executes "nc" with its stdout
associated with the pty.

[toc] | [prev] | [next] | [standalone]

#79273

On 2014-10-18 17:55, Nobody wrote:
> On Fri, 17 Oct 2014 12:38:54 +0100, Empty Account wrote:
> 
> > I am using netcat to listen to a port and python to read stdin
> > and print to the console.
> > 
> > nc -l 2003 | python print_metrics.py
> > 
> > sys.stdin.flush() doesn’t seem to flush stdin,
> 
> You can't "flush" an input stream.

You can't flush it, but you can make it unbuffered.  You can either
force python to use unbuffered stdio:

  python -u myfile.py

or you can get an unbuffered handle to the file

 import os, sys
 buffer_size = 1
 new_stdin = os.fdopen(sys.stdin.fileno(), 'r', buffer_size)
 for c in new_stdin:
   do_something(c)

though based on my reading, the first method works with both Py2
and Py3k while the second method doesn't reliably work in Py3k.

-tkc

[toc] | [prev] | [next] | [standalone]

#79281

On Sat, 18 Oct 2014 12:32:07 -0500, Tim Chase wrote:

> On 2014-10-18 17:55, Nobody wrote:
>> On Fri, 17 Oct 2014 12:38:54 +0100, Empty Account wrote:
>> 
>> > I am using netcat to listen to a port and python to read stdin and
>> > print to the console.
>> > 
>> > nc -l 2003 | python print_metrics.py
>> > 
>> > sys.stdin.flush() doesn’t seem to flush stdin,
>> 
>> You can't "flush" an input stream.
> 
> You can't flush it, but you can make it unbuffered.  You can either force
> python to use unbuffered stdio:

[snipped]

None of this helps in any way, as it's not the behaviour of the Python
script which is causing the problem, but that "nc" is (probably) buffering
its output, so the data isn't passed to the OS (let alone to the Python
script) in a timely manner.

Once the "nc" process actually write()s the data to its standard
output (i.e. desriptor 1, not the "stdout" FILE*), it will be available to
the Python script immediately thereafter without requiring any low-level
tweaks.

[toc] | [prev] | [next] | [standalone]

#79283

On Sat, Oct 18, 2014 at 6:11 PM, Nobody <nobody@nowhere.invalid> wrote:
> On Sat, 18 Oct 2014 12:32:07 -0500, Tim Chase wrote:
>
>> On 2014-10-18 17:55, Nobody wrote:
>>> On Fri, 17 Oct 2014 12:38:54 +0100, Empty Account wrote:
>>>
>>> > I am using netcat to listen to a port and python to read stdin and
>>> > print to the console.
>>> >
>>> > nc -l 2003 | python print_metrics.py
>>> >
>>> > sys.stdin.flush() doesn’t seem to flush stdin,
>>>
>>> You can't "flush" an input stream.
>>
>> You can't flush it, but you can make it unbuffered.  You can either force
>> python to use unbuffered stdio:
>
> [snipped]
>
> None of this helps in any way, as it's not the behaviour of the Python
> script which is causing the problem, but that "nc" is (probably) buffering
> its output, so the data isn't passed to the OS (let alone to the Python
> script) in a timely manner.

Agreed.

> Once the "nc" process actually write()s the data to its standard
> output (i.e. desriptor 1, not the "stdout" FILE*)
I'm not sure why you're excluding stdout, but even if nc is using
filedes 1 instead of FILE * stdout, isn't it kind of irrelevant?

> it will be available to
> the Python script immediately thereafter without requiring any low-level
> tweaks.
Which, on a pipe, generally means either the buffer filled and needed
to be passed along to make room, or the process exited.

I'd probably rewrite just enough nc in Python to make it so you don't
need to depend on a pipe (example:
http://stromberg.dnsalias.org/~strombrg/pnetcat.html), but if you're
on *ix you could try
http://ftp.sunet.se/pub/usenet/ftp.uu.net/comp.sources.unix/volume23/pty/
in an effort to persuade nc to think that it's on a tty and hence
should output line buffered data instead of block buffered - despite
being on a pipe in reality.

HTH.

[toc] | [prev] | [next] | [standalone]

#79284

On Sat, Oct 18, 2014 at 6:34 PM, Dan Stromberg <drsalists@gmail.com> wrote:
>> Once the "nc" process actually write()s the data to its standard
>> output (i.e. desriptor 1, not the "stdout" FILE*)
> I'm not sure why you're excluding stdout, but even if nc is using
> filedes 1 instead of FILE * stdout, isn't it kind of irrelevant?

On further reflection, isn't it stdio that does the varied buffering,
and filedes 1 that's always unbuffered?  IOW, the OP might wish nc was
using 1, but it probably can't be given what they're seeing.

[toc] | [prev] | [next] | [standalone]

#79409

On Sat, 18 Oct 2014 18:42:00 -0700, Dan Stromberg wrote:

> On Sat, Oct 18, 2014 at 6:34 PM, Dan Stromberg <drsalists@gmail.com> wrote:
>>> Once the "nc" process actually write()s the data to its standard
>>> output (i.e. desriptor 1, not the "stdout" FILE*)
>> I'm not sure why you're excluding stdout, but even if nc is using
>> filedes 1 instead of FILE * stdout, isn't it kind of irrelevant?
> 
> On further reflection, isn't it stdio that does the varied buffering,
> and filedes 1 that's always unbuffered?  IOW, the OP might wish nc was
> using 1, but it probably can't be given what they're seeing.

Yes. stdio does buffering. Writing to stdout stores data in a buffer; that
data should eventually be written to descriptor 1, although perhaps not
until immediately prior to termination.

Which is probably the cause of the OP's problem.

If it is, using a pseudo-tty would probably fix it. At startup,
stdin and stdout are line-buffered if they are associated with a tty and
fully-buffered otherwise (file, pipe, ...); stderr is unbuffered.

At least, this is the case on Unix and Windows. The exact requirements of
the C standard are:

	As initially opened, the standard error stream is not fully
	buffered; the standard input and standard output streams are
	fully buffered if and only if the stream can be determined not
	to refer to an interactive device.

[toc] | [prev] | [next] | [standalone]

#79410

On Tue, Oct 21, 2014 at 7:49 PM, Nobody <nobody@nowhere.invalid> wrote:
> On Sat, 18 Oct 2014 18:42:00 -0700, Dan Stromberg wrote:
>
>> On Sat, Oct 18, 2014 at 6:34 PM, Dan Stromberg <drsalists@gmail.com> wrote:
>>>> Once the "nc" process actually write()s the data to its standard
>>>> output (i.e. desriptor 1, not the "stdout" FILE*)
>>> I'm not sure why you're excluding stdout, but even if nc is using
>>> filedes 1 instead of FILE * stdout, isn't it kind of irrelevant?
>>
>> On further reflection, isn't it stdio that does the varied buffering,
>> and filedes 1 that's always unbuffered?  IOW, the OP might wish nc was
>> using 1, but it probably can't be given what they're seeing.
>
> Yes. stdio does buffering. Writing to stdout stores data in a buffer; that
> data should eventually be written to descriptor 1, although perhaps not
> until immediately prior to termination.
>
> Which is probably the cause of the OP's problem.

Huh.  And here I thought I demonstrated elsewhere in this thread, that
the buffering between nc and python didn't appear to be the problem.

'found it, here it is again:

If I run the following in one tty:
    nc -l localhost 9000 | /tmp/z

...where /tmp/z has just:
    #!/usr/bin/python3

    import sys

    for line in sys.stdin.buffer:
        print(line)

And then run the following in another tty on the same computer:
    while read line; do echo $line; sleep 1; done < /etc/passwd | nc
localhost 9000

...then everything acts line buffered, or perhaps even character
buffered (the two are pretty indistinguishable in this test).  What I
see is my /etc/passwd file popping out of the nc -l side, one line at
a time, each line one second apart.

I suppose this suggests that it's the client that's sending TCP data
that is buffering.

That, or we're using two different versions of netcat (there are at
least two available).

[toc] | [prev] | [next] | [standalone]

#79274

On 2014-10-18 17:55, Nobody wrote:
> On Fri, 17 Oct 2014 12:38:54 +0100, Empty Account wrote:
>
>> I am using netcat to listen to a port and python to read stdin and print
>> to the console.
>>
>> nc -l 2003 | python print_metrics.py
>>
>> sys.stdin.flush() doesn’t seem to flush stdin,
>
> You can't "flush" an input stream.
>
[snip]

Flushing an input stream means (or could mean) discarding any data
that's currently in the buffer.

[toc] | [prev] | [next] | [standalone]

#79279

On 18Oct2014 17:55, Nobody <nobody@nowhere.invalid> wrote:
>On Fri, 17 Oct 2014 12:38:54 +0100, Empty Account wrote:
>> I am using netcat to listen to a port and python to read stdin and print
>> to the console.
>>
>> nc -l 2003 | python print_metrics.py
>>
>> sys.stdin.flush() doesn’t seem to flush stdin,
>
>You can't "flush" an input stream.

Sure you can. Most streams are read through an API which buffers. That buffer 
can be discarded.

I'm not sure it is what the OP needs, but it is not a nonsensical idea.

Cheers,
Cameron Simpson <cs@zip.com.au>

I've seen things you people wouldn't believe.  Attack ships on fire off the
shoulder of Orion. I've watched C-beams glitter in the dark near the
Tannhauser Gate.  All these memories will be lost in time, like tears in rain.
         - Roy Baty, _Blade Runner_

[toc] | [prev] | [next] | [standalone]

#79280

On 10/18/2014 5:01 PM, Cameron Simpson wrote:
> On 18Oct2014 17:55, Nobody <nobody@nowhere.invalid> wrote:
>> On Fri, 17 Oct 2014 12:38:54 +0100, Empty Account wrote:
>>> I am using netcat to listen to a port and python to read stdin and print
>>> to the console.
>>>
>>> nc -l 2003 | python print_metrics.py
>>>
>>> sys.stdin.flush() doesn’t seem to flush stdin,
>>
>> You can't "flush" an input stream.

> Sure you can.

You are collectively confusing three different meaning of 'flush'. 
Python and its docs are the best authority on what python can and cannot 
do.  One can "call .flush() on an imput stream" (meaning 1).

 >>> import sys
 >>> sys.stdin.flush()
 >>>

However, one cannot "empty the steam buffer by calling .flush()" 
(meaning 2).

" class IOBase
...
flush()
     Flush the write buffers of the stream if applicable. This does 
nothing for read-only and non-blocking streams."

> Most streams are read through an API which buffers. That
> buffer can be discarded.

But one can "empty and discard the buffer" (meaning 3) with
stream.read()

And, of course, an 'input stream' 'down here' is an 'output stream' 'up 
there, where ever' and one can 'flush' pending output into the stream so 
that it can be read here.

-- 
Terry Jan Reedy

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web