Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.003 X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'example:': 0.03; 'subject:not': 0.03; 'causing': 0.04; 'output': 0.05; 'python)': 0.05; 'true,': 0.05; '(using': 0.07; 'everybody,': 0.07; "subject:' ": 0.07; 'occasionally': 0.09; 'parsed': 0.09; 'wrapper': 0.09; 'cc:addr:python-list': 0.11; 'python': 0.11; 'def': 0.12; '/dev/null': 0.16; '2.7.3': 0.16; '24,': 0.16; 'problem).': 0.16; 'stderr': 0.16; 'stdout': 0.16; 'unlikely': 0.16; 'using,': 0.16; 'weekend.': 0.16; 'exception': 0.16; 'topics': 0.16; 'wrote:': 0.18; 'code.': 0.18; 'library': 0.18; 'written': 0.21; 'command': 0.22; 'import': 0.22; 'email addr:gmail.com>': 0.22; 'shell': 0.22; 'cc:addr:python.org': 0.22; 'print': 0.22; 'error': 0.23; 'parse': 0.24; 'cheers,': 0.24; 'cc:2**0': 0.24; '>': 0.26; '(see': 0.26; 'second': 0.26; 'header:In-Reply-To:1': 0.27; 'function': 0.29; 'chris': 0.29; 'am,': 0.29; "doesn't": 0.30; 'message-id:@mail.gmail.com': 0.30; 'code': 0.31; 'easier': 0.31; '(although': 0.31; 'large.': 0.31; 'occurs': 0.31; 'pipe': 0.31; 'linux': 0.33; 'url:python': 0.33; 'running': 0.33; 'could': 0.34; 'subject:with': 0.35; 'basic': 0.35; "can't": 0.35; 'common': 0.35; 'something': 0.35; 'but': 0.35; 'received:google.com': 0.35; 'version': 0.36; 'really': 0.36; 'url:org': 0.36; 'searching': 0.37; 'wrong': 0.37; 'two': 0.37; 'skip:& 10': 0.38; 'generic': 0.38; 'url:library': 0.38; 'rather': 0.38; 'skip:& 20': 0.39; "couldn't": 0.39; 'skip:u 10': 0.60; 'above,': 0.60; 'commands': 0.60; 'full': 0.61; 'new': 0.61; "you're": 0.61; 'first': 0.61; 'complete': 0.62; 'such': 0.63; 'provide': 0.64; 'occur': 0.65; 'relatively': 0.65; 'to:addr:gmail.com': 0.65; 'here': 0.66; 'between': 0.67; 'approaches': 0.68; 'obvious': 0.74; 'programs,': 0.74; 'day': 0.76; 'as:': 0.81; 'lastly,': 0.84; 'memorial': 0.84; 'redirecting': 0.84; 'sender:addr:chris': 0.84; 'streams': 0.84; 'subject:True': 0.91; 'url:latest': 0.91; 'whereas': 0.91; '2013': 0.98 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rebertia.com; s=google; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=5xLtR3J3/QJ7AV7BWHwqrlbKc4sLciYL9Kq+AvuMYsw=; b=AZPBCgBVz988dBHToYw2VMUBXukslgPF2PzykSOUeFB+Bg8nPxolPHVc8dN4mjws6S TfIS/03OZnGP1nHF2m0kgye7r6dUdgsr9TpCTpKobiXhJAhlY+kMKqbsFuTT9d9ul8yQ d1KX/19byt2sWavVlwzQhWKDD6aGGxfOUiPlU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :x-gm-message-state; bh=5xLtR3J3/QJ7AV7BWHwqrlbKc4sLciYL9Kq+AvuMYsw=; b=KSzs/qTk2IG9wmf9LOTN5zFrxoJ4J3RjAHg5t2Gtgv1wTjccFQ5XQFAgu89LJBs2O/ 6kOAZJxOdKo+DlQzyRIiDuZXG++XaNqUw1F6wj5HaLVM61ITVyF7U3uYwWxRverrA2V6 gDD4BglzV68XX2xMAd6XMWJEsCKEV7HN69CGLexHx4nE56arUkIoCABeBafMDhS13qMO la33eKNIpLf5Bh81O4E5JtERxm1tJAZiSOF7h/QhlqozHCI1ZE+ajPVb3d4zcYSU+nYh EFQ2D23igVT5AblAWsj9wRnjq3CAuIK/17CBKkt3D9hsCzEVd8AQKqdKuh3vsNgSmu9f GNyA== MIME-Version: 1.0 X-Received: by 10.50.120.68 with SMTP id la4mr3725904igb.49.1369602319120; Sun, 26 May 2013 14:05:19 -0700 (PDT) Sender: chris@rebertia.com In-Reply-To: <062f557e-8e1a-4efb-9178-7d685b47a834@googlegroups.com> References: <062f557e-8e1a-4efb-9178-7d685b47a834@googlegroups.com> Date: Sun, 26 May 2013 14:05:19 -0700 X-Google-Sender-Auth: X22Rj9wZOxZv5Ea2UoJ5vxHM-kk Subject: Re: Piping processes works with 'shell = True' but not otherwise. From: Chris Rebert To: Luca Cerone Content-Type: multipart/alternative; boundary=047d7ba97978b7119804dda5639b X-Gm-Message-State: ALoCoQnpqnB1eEsqOetCvW+NSnfHJnqDvyfx12TV+b0HBami+VGn+hcy5EG5nIycAqUsb1b7yfVw Cc: Python X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 188 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1369602328 news.xs4all.nl 15951 [2001:888:2000:d::a6]:47391 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:46122 --047d7ba97978b7119804dda5639b Content-Type: text/plain; charset=UTF-8 On May 24, 2013 7:06 AM, "Luca Cerone" wrote: > > Hi everybody, > I am new to the group (and relatively new to Python) > so I am sorry if this issues has been discussed (although searching for topics in the group I couldn't find a solution to my problem). > > I am using Python 2.7.3 to analyse the output of two 3rd parties programs that can be launched in a linux shell as: > > program1 | program2 > > To do this I have written a function that pipes program1 and program2 (using subprocess.Popen) and the stdout of the subprocess, and a function that parses the output: > > A basic example: > > from subprocess import Popen, STDOUT, PIPE > def run(): > p1 = Popen(['program1'], stdout = PIPE, stderr = STDOUT) > p2 = Popen(['program2'], stdin = p1.stdout, stdout = PIPE, stderr = STDOUT) Could you provide the *actual* commands you're using, rather than the generic "program1" and "program2" placeholders? It's *very* common for people to get the tokenization of a command line wrong (see the Note box in http://docs.python.org/2/library/subprocess.html#subprocess.Popen for some relevant advice). > p1.stdout.close() > return p2.stdout > > > def parse(out): > for row in out: > print row > #do something else with each line > out.close() > return parsed_output > > > # main block here > > pout = run() > > parsed = parse(pout) > > #--- END OF PROGRAM ----# > > I want to parse the output of 'program1 | program2' line by line because the output is very large. > > When running the code above, occasionally some error occurs (IOERROR: [Errno 0]). Could you provide the full & complete error message and exception traceback? > However this error doesn't occur if I code the run() function as: > > def run(): > p = Popen('program1 | program2', shell = True, stderr = STDOUT, stdout = PIPE) > return p.stdout > > I really can't understand why the first version causes errors, while the second one doesn't. > > Can you please help me understanding what's the difference between the two cases? One obvious difference between the 2 approaches is that the shell doesn't redirect the stderr streams of the programs, whereas you /are/ redirecting the stderrs to stdout in the non-shell version of your code. But this is unlikely to be causing the error you're currently seeing. You may also want to provide /dev/null as p1's stdin, out of an abundance of caution. Lastly, you may want to consider using a wrapper library such as http://plumbum.readthedocs.org/en/latest/ , which makes it easier to do pipelining and other such "fancy" things with subprocesses, while still avoiding the many perils of the shell. Cheers, Chris -- Be patient; it's Memorial Day weekend. --047d7ba97978b7119804dda5639b Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

On May 24, 2013 7:06 AM, "Luca Cerone" <luca.cerone@gmail.com> wrote:
>
> Hi everybody,
> I am new to the group (and relatively new to Python)
> so I am sorry if this issues has been discussed (although searching fo= r topics in the group I couldn't find a solution to my problem).
>
> I am using Python 2.7.3 to analyse the output of two 3rd parties progr= ams that can be launched in a linux shell as:
>
> =C2=A0program1 | program2
>
> To do this I have written a function that pipes program1 and program2 = (using subprocess.Popen) and the stdout of the subprocess, and a function t= hat parses the output:
>
> A basic example:
>
> from subprocess import Popen, STDOUT, PIPE
> def run():
> =C2=A0 p1 =3D Popen(['program1'], stdout =3D PIPE, stderr =3D = STDOUT)
> =C2=A0 p2 =3D Popen(['program2'], stdin =3D p1.stdout, stdout = =3D PIPE, stderr =3D STDOUT)

Could you provide the *actual* commands you're using, ra= ther than the generic "program1" and "program2" placeho= lders? It's *very* common for people to get the tokenization of a comma= nd line wrong (see the Note box in http://docs.python.org/2/library/subp= rocess.html#subprocess.Popen for some relevant advice).

> =C2=A0 p1.stdout.close()
> =C2=A0 return p2.stdout
>
>
> def parse(out):
> =C2=A0 for row in out:
> =C2=A0 =C2=A0 print row
> =C2=A0 =C2=A0 #do something else with each line
> =C2=A0 out.close()
> =C2=A0 return parsed_output
>
>
> # main block here
>
> pout =3D run()
>
> parsed =3D parse(pout)
>
> #--- END OF PROGRAM ----#
>
> I want to parse the output of 'program1 | program2' line by li= ne because the output is very large.
>
> When running the code above, occasionally some error occurs (IOERROR: = [Errno 0]).

Could you provide the full & complete error message and = exception traceback?

> However this error doesn't occur if I code the run(= ) function as:
>
> def run():
> =C2=A0 p =3D Popen('program1 | program2', shell =3D True, stde= rr =3D STDOUT, stdout =3D PIPE)
> =C2=A0 return p.stdout
>
> I really can't understand why the first version causes errors, whi= le the second one doesn't.
>
> Can you please help me understanding what's the difference between= the two cases?

One obvious difference between the 2 approaches is that the = shell doesn't redirect the stderr streams of the programs, whereas you = /are/ redirecting the stderrs to stdout in the non-shell version of your co= de. But this is unlikely to be causing the error you're currently seein= g.

You may also want to provide /dev/null as p1's stdin, ou= t of an abundance of caution.

Lastly, you may want to consider using a wrapper library suc= h as http://plumbum.r= eadthedocs.org/en/latest/ , which makes it easier to do pipelining and = other such "fancy" things with subprocesses, while still avoiding= the many perils of the shell.

Cheers,
Chris
--
Be patient; it's Memorial Day weekend.

--047d7ba97978b7119804dda5639b--