Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!news.glorb.com!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.001 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'output': 0.05; 'received:169.254.1': 0.05; '"__main__":': 0.09; '[],': 0.09; '__name__': 0.09; 'executes': 0.09; 'newline': 0.09; 'pointers': 0.09; 'received:10.14': 0.09; 'seemed': 0.09; 'try:': 0.09; 'python': 0.11; 'def': 0.12; 'wrote': 0.14; '%r"': 0.16; '0.1)': 0.16; '2.6.4': 0.16; 'attaching': 0.16; 'fcntl': 0.16; 'googling': 0.16; 'pty': 0.16; 'simplest': 0.16; 'stdout': 0.16; 'true:': 0.16; 'fix': 0.17; 'thanks,': 0.17; 'select': 0.22; 'import': 0.22; 'to:name:python-list@python.org': 0.22; 'print': 0.22; 'environment': 0.24; "i've": 0.25; 'this:': 0.26; 'testing': 0.29; 'correct': 0.29; 'external': 0.29; 'mode': 0.30; 'specified': 0.30; '(which': 0.31; 'getting': 0.31; 'bunch': 0.31; 'controlled': 0.31; 'enabled': 0.31; 'anyone': 0.31; 'class': 0.32; 'received:169.254': 0.32; 'cases': 0.33; 'skip:# 10': 0.33; 'noticed': 0.34; 'skip:_ 10': 0.34; "i'd": 0.34; 'could': 0.34; 'subject:with': 0.35; 'except': 0.35; 'skip:u 20': 0.35; 'test': 0.35; 'done': 0.36; 'charset:us-ascii': 0.36; 'hi,': 0.36; 'received:169': 0.37; 'subject:New': 0.37; 'wrong': 0.37; 'received:10': 0.37; 'skip:o 20': 0.38; 'needed': 0.38; 'to:addr :python-list': 0.38; 'fact': 0.38; 'does': 0.39; 'environment.': 0.39; 'to:addr:python.org': 0.39; 'even': 0.60; 'skip:u 10': 0.60; 'break': 0.61; 'conversion': 0.61; 'new': 0.61; 'simple': 0.61; 'kept': 0.65; 'hours': 0.66; 'message).': 0.84; 'skip:| 10': 0.84 From: Jonathan Harden To: "python-list@python.org" Subject: New line conversion with Popen attached to a pty Thread-Topic: New line conversion with Popen attached to a pty Thread-Index: AQHObZyym1MomopBoEuTo4mCJc1l7w== Date: Thu, 20 Jun 2013 10:20:20 +0000 Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.16.22.176] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Spam-Score: -2.9 (--) X-Spam-Report: Spam detection software, running on the system "caroline.mpc.local", has identified this incoming email as possible spam. The original message has been attached to this so you can view it (if it isn't spam) or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Hi, We have a class which executes external processes in a controlled environment and does "things" specified by the client program with each line of output. To do this we have been attaching stdout from the subprocess.Popen to a pseudo terminal (pty) made with pty.openempty and opened with os.fdopen. I noticed that we kept getting a bunch of extra new line characters. [...] Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Mailman-Approved-At: Fri, 21 Jun 2013 13:00:12 +0200 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 89 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1371812413 news.xs4all.nl 15865 [2001:888:2000:d::a6]:45892 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:48856 Hi,=0A= =0A= We have a class which executes external processes in a controlled environme= nt and does "things" specified by the client program with each line of outp= ut. To do this we have been attaching stdout from the subprocess.Popen to a= pseudo terminal (pty) made with pty.openempty and opened with os.fdopen. I= noticed that we kept getting a bunch of extra new line characters. =0A= =0A= This is all using python 2.6.4 in a centos6 environment.=0A= =0A= After some investigation I realised we needed to use universal_newline supp= ort so I enabled it for the Popen and specified the mode in the fdopen to b= e rU. Things still seemed to be coming out wrong so I wrote up a test progr= am boiling it down to the simplest cases (which is at the end of this messa= ge). The output I was testing was this:=0A= =0A= Fake\r\nData\r\n=0A= as seen through hexdump -C:=0A= =0A= > hexdump -C output.txt =0A= 00000000 46 61 6b 65 0d 0a 44 61 74 61 0d 0a |Fake..Data..|= =0A= 0000000c=0A= =0A= Now if I do a simple subprocess.Popen and set the stdout to subprocess.PIPE= , then do p.stdout.read() I get the correct output of=0A= =0A= Fake\nData\n=0A= =0A= When do the Popen attached to a pty I end up with=0A= =0A= Fake\n\nData\n\n=0A= =0A= Does anyone know why the newline conversion would be incorrect, and what I = could do to fix it? In fact if anyone even has any pointers to where this m= ight be going wrong I'd be very helpful, I've done a lot of hours of fiddli= ng with this and googling to no avail.=0A= =0A= Thanks,=0A= =0A= Jonathan=0A= =0A= #!/usr/bin/env python2.6.4=0A= import os=0A= import pty=0A= import subprocess=0A= import select=0A= import fcntl=0A= =0A= class TestRead(object):=0A= =0A= def __init__(self):=0A= super(TestRead, self).__init__()=0A= self.outputPipe()=0A= self.outputPty()=0A= =0A= def outputPipe(self):=0A= p1 =3D subprocess.Popen(=0A= ("/bin/cat", "output.txt"),=0A= stdout=3Dsubprocess.PIPE,=0A= universal_newlines=3DTrue=0A= )=0A= print "1: %r" % p1.stdout.read()=0A= =0A= def outputPty(self):=0A= outMaster, outSlave =3D pty.openpty()=0A= fcntl.fcntl(outMaster, fcntl.F_SETFL, os.O_NONBLOCK)=0A= =0A= p2 =3D subprocess.Popen(=0A= ("/bin/cat", "output.txt"),=0A= stdout=3DoutSlave,=0A= universal_newlines=3DTrue=0A= )=0A= =0A= with os.fdopen(outMaster, 'rU') as pty_stdout:=0A= while True:=0A= try:=0A= rfds, _, _ =3D select.select([pty_stdout], [], [], 0.1)= =0A= break=0A= except select.error:=0A= continue=0A= =0A= for fd in rfds:=0A= buf =3D pty_stdout.read()=0A= print "2: %r" % buf=0A= =0A= if __name__ =3D=3D "__main__":=0A= t =3D TestRead()=