Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #55820 > unrolled thread

Reassign or discard Popen().stdout from a server process

Back to article view | Back to comp.lang.python

Contents

  Reassign or discard Popen().stdout from a server process "John O'Hagan" <mail@johnohagan.com> - 2011-02-01 08:30 +0000
    Re: Reassign or discard Popen().stdout from a server process "John O'Hagan" <research@johnohagan.com> - 2011-02-04 15:48 +0000
      Re: Reassign or discard Popen().stdout from a server process "John O'Hagan" <research@johnohagan.com> - 2011-02-10 08:35 +0000
      Re: Reassign or discard Popen().stdout from a server process Nobody <nobody@nowhere.com> - 2011-02-09 05:24 +0000
    Re: Reassign or discard Popen().stdout from a server process Nobody <nobody@nowhere.com> - 2011-02-03 09:48 +0000

#55820 — Reassign or discard Popen().stdout from a server process

I'm starting a server process as a subprocess. Startup is slow and 
unpredictable (around 3-10 sec), so I'm reading from its stdout until I get a 
line that tells me it's ready before proceeding, in simplified form:

import subprocess
proc = subprocess.Popen(['server', 'args'], stdout=subprocess.PIPE)
while proc.stdout.readline() != "Ready.\n":
    pass

Now I can start communicating with the server, but I eventually realised that 
as I'm no longer reading stdout, the pipe buffer will fill up with output from 
the server and before long it blocks and the server stops working. 

I can't keep reading because that will block - there won't be any more output 
until I send some input, and I don't want it in any case.

To try to fix this I added:

proc.stdout = os.path.devnull

which has the effect of stopping the server from failing, but I'm not convinced 
it's doing what I think it is. If I replace devnull in the above line with a 
real file, it stays empty although I know there is more output, which makes me 
think it hasn't really worked. 

Simply closing stdout also seems to stop the crashes, but doesn't that mean 
it's still being written to, but the writes are just silently failing? In 
either case I'm wary of more elusive bugs arising from misdirected stdout.

Is it possible to re-assign the stdout of a subprocess after it has started? 
Or just close it? What's the right way to read stdout up to a given line, then 
discard the rest?

Thanks,

john

[toc] | [next] | [standalone]

#55857

On Thu, 3 Feb 2011, Nobody wrote:
> On Tue, 01 Feb 2011 08:30:19 +0000, John O'Hagan wrote:
> > I can't keep reading because that will block - there won't be any more
> > output until I send some input, and I don't want it in any case.
> > 
> > To try to fix this I added:
> > 
> > proc.stdout = os.path.devnull
> > 
> > which has the effect of stopping the server from failing, but I'm not
> > convinced it's doing what I think it is.
> 
> It isn't. os.path.devnull is a string, not a file. But even if you did:
> 
> 	proc.stdout = open(os.path.devnull, 'w')
> 
> that still wouldn't work.

As mentioned earlier in the thread, I did in fact use open(), this was a typo,
 [...]
> > Is it possible to re-assign the stdout of a subprocess after it has
> > started?
> 
> No.
> 
> > Or just close it? What's the right way to read stdout up to a given
> > line, then discard the rest?
> 
> If the server can handle the pipe being closed, go with that. Otherwise,
> options include redirecting stdout to a file and running "tail -f" on the
> file from within Python, or starting a thread or process whose sole
> function is to read and discard the server's output.

Thanks, that's all clear now. 

But I'm still a little curious as to why even unsuccessfully attempting to 
reassign stdout seems to stop the pipe buffer from filling up.

John 

[toc] | [prev] | [next] | [standalone]

#55926

On Wed, 9 Feb 2011, Nobody wrote:
> On Fri, 04 Feb 2011 15:48:55 +0000, John O'Hagan wrote:
> > But I'm still a little curious as to why even unsuccessfully attempting
> > to reassign stdout seems to stop the pipe buffer from filling up.
> 
> It doesn't. If the server continues to run, then it's ignoring/handling
> both SIGPIPE and the EPIPE error. Either that, or another process has the
> read end of the pipe open (so no SIGPIPE/EPIPE), and the server is using
> non-blocking I/O or select() so that it doesn't block writing its
> diagnostic messages.

The server fails with stdout=PIPE if I don't keep reading it, but doesn't fail 
if I do stdout=anything (I've tried files, strings, integers, and None) soon 
after starting the process, without any other changes. How is that consistent 
with either of the above conditions? I'm sure you're right, I just don't 
understand.

Regards,

John

[toc] | [prev] | [next] | [standalone]

#56146

On Fri, 04 Feb 2011 15:48:55 +0000, John O'Hagan wrote:

> But I'm still a little curious as to why even unsuccessfully attempting to
> reassign stdout seems to stop the pipe buffer from filling up.

It doesn't. If the server continues to run, then it's ignoring/handling
both SIGPIPE and the EPIPE error. Either that, or another process has the
read end of the pipe open (so no SIGPIPE/EPIPE), and the server is using
non-blocking I/O or select() so that it doesn't block writing its
diagnostic messages.

[toc] | [prev] | [next] | [standalone]

#55967

On Tue, 01 Feb 2011 08:30:19 +0000, John O'Hagan wrote:

> I can't keep reading because that will block - there won't be any more
> output until I send some input, and I don't want it in any case.
> 
> To try to fix this I added:
> 
> proc.stdout = os.path.devnull
> 
> which has the effect of stopping the server from failing, but I'm not
> convinced it's doing what I think it is.

It isn't. os.path.devnull is a string, not a file. But even if you did:

	proc.stdout = open(os.path.devnull, 'w')

that still wouldn't work.

> If I replace devnull in the above line with a real file, it stays empty
> although I know there is more output, which makes me think it hasn't
> really worked. 

It hasn't.

> Simply closing stdout also seems to stop the crashes, but doesn't that mean 
> it's still being written to, but the writes are just silently failing? In 
> either case I'm wary of more elusive bugs arising from misdirected stdout.

If you close proc.stdout, the next time the server writes to its stdout,
it will receive SIGPIPE or, if it catches that, the write will fail with
EPIPE (write on pipe with no readers). It's up to the server how it deals
with that.

> Is it possible to re-assign the stdout of a subprocess after it has started? 

No.

> Or just close it? What's the right way to read stdout up to a given
> line, then discard the rest?

If the server can handle the pipe being closed, go with that. Otherwise,
options include redirecting stdout to a file and running "tail -f" on the
file from within Python, or starting a thread or process whose sole
function is to read and discard the server's output.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web