Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #48856

New line conversion with Popen attached to a pty

Show key headers only | View raw

Hi,

We have a class which executes external processes in a controlled environment and does "things" specified by the client program with each line of output. To do this we have been attaching stdout from the subprocess.Popen to a pseudo terminal (pty) made with pty.openempty and opened with os.fdopen. I noticed that we kept getting a bunch of extra new line characters. 

This is all using python 2.6.4 in a centos6 environment.

After some investigation I realised we needed to use universal_newline support so I enabled it for the Popen and specified the mode in the fdopen to be rU. Things still seemed to be coming out wrong so I wrote up a test program boiling it down to the simplest cases (which is at the end of this message). The output I was testing was this:

Fake\r\nData\r\n
as seen through hexdump -C:

> hexdump -C output.txt 
00000000  46 61 6b 65 0d 0a 44 61  74 61 0d 0a              |Fake..Data..|
0000000c

Now if I do a simple subprocess.Popen and set the stdout to subprocess.PIPE, then do p.stdout.read() I get the correct output of

Fake\nData\n

When do the Popen attached to a pty I end up with

Fake\n\nData\n\n

Does anyone know why the newline conversion would be incorrect, and what I could do to fix it? In fact if anyone even has any pointers to where this might be going wrong I'd be very helpful, I've done a lot of hours of fiddling with this and googling to no avail.

Thanks,

Jonathan

#!/usr/bin/env python2.6.4
import os
import pty
import subprocess
import select
import fcntl

class TestRead(object):

    def __init__(self):
        super(TestRead, self).__init__()
        self.outputPipe()
        self.outputPty()

    def outputPipe(self):
        p1 = subprocess.Popen(
            ("/bin/cat", "output.txt"),
            stdout=subprocess.PIPE,
            universal_newlines=True
        )
        print "1: %r" % p1.stdout.read()

    def outputPty(self):
        outMaster, outSlave = pty.openpty()
        fcntl.fcntl(outMaster, fcntl.F_SETFL, os.O_NONBLOCK)

        p2 = subprocess.Popen(
            ("/bin/cat", "output.txt"),
            stdout=outSlave,
            universal_newlines=True
        )

        with os.fdopen(outMaster, 'rU') as pty_stdout:
            while True:
                try:
                    rfds, _, _ = select.select([pty_stdout], [], [], 0.1)
                    break
                except select.error:
                    continue

            for fd in rfds:
                buf = pty_stdout.read()
                print "2: %r" % buf

if __name__ == "__main__":
    t = TestRead()

Back to comp.lang.python | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

New line conversion with Popen attached to a pty Jonathan Harden <jonathan-h@moving-picture.com> - 2013-06-20 10:20 +0000
  Re: New line conversion with Popen attached to a pty jfharden@gmail.com - 2013-06-21 04:14 -0700

csiph-web