Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #46122

Re: Piping processes works with 'shell = True' but not otherwise.

Show all headers | View raw

[Multipart message — attachments visible in raw view] - view raw

On May 24, 2013 7:06 AM, "Luca Cerone" <luca.cerone@gmail.com> wrote:
>
> Hi everybody,
> I am new to the group (and relatively new to Python)
> so I am sorry if this issues has been discussed (although searching for
topics in the group I couldn't find a solution to my problem).
>
> I am using Python 2.7.3 to analyse the output of two 3rd parties programs
that can be launched in a linux shell as:
>
>  program1 | program2
>
> To do this I have written a function that pipes program1 and program2
(using subprocess.Popen) and the stdout of the subprocess, and a function
that parses the output:
>
> A basic example:
>
> from subprocess import Popen, STDOUT, PIPE
> def run():
>   p1 = Popen(['program1'], stdout = PIPE, stderr = STDOUT)
>   p2 = Popen(['program2'], stdin = p1.stdout, stdout = PIPE, stderr =
STDOUT)

Could you provide the *actual* commands you're using, rather than the
generic "program1" and "program2" placeholders? It's *very* common for
people to get the tokenization of a command line wrong (see the Note box in
http://docs.python.org/2/library/subprocess.html#subprocess.Popen for some
relevant advice).

>   p1.stdout.close()
>   return p2.stdout
>
>
> def parse(out):
>   for row in out:
>     print row
>     #do something else with each line
>   out.close()
>   return parsed_output
>
>
> # main block here
>
> pout = run()
>
> parsed = parse(pout)
>
> #--- END OF PROGRAM ----#
>
> I want to parse the output of 'program1 | program2' line by line because
the output is very large.
>
> When running the code above, occasionally some error occurs (IOERROR:
[Errno 0]).

Could you provide the full & complete error message and exception traceback?

> However this error doesn't occur if I code the run() function as:
>
> def run():
>   p = Popen('program1 | program2', shell = True, stderr = STDOUT, stdout
= PIPE)
>   return p.stdout
>
> I really can't understand why the first version causes errors, while the
second one doesn't.
>
> Can you please help me understanding what's the difference between the
two cases?

One obvious difference between the 2 approaches is that the shell doesn't
redirect the stderr streams of the programs, whereas you /are/ redirecting
the stderrs to stdout in the non-shell version of your code. But this is
unlikely to be causing the error you're currently seeing.

You may also want to provide /dev/null as p1's stdin, out of an abundance
of caution.

Lastly, you may want to consider using a wrapper library such as
http://plumbum.readthedocs.org/en/latest/ , which makes it easier to do
pipelining and other such "fancy" things with subprocesses, while still
avoiding the many perils of the shell.

Cheers,
Chris
--
Be patient; it's Memorial Day weekend.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Piping processes works with 'shell = True' but not otherwise. Luca Cerone <luca.cerone@gmail.com> - 2013-05-24 07:04 -0700
  Re: Piping processes works with 'shell = True' but not otherwise. Luca Cerone <luca.cerone@gmail.com> - 2013-05-26 03:31 -0700
  Re: Piping processes works with 'shell = True' but not otherwise. Chris Rebert <clp2@rebertia.com> - 2013-05-26 14:05 -0700
    Re: Piping processes works with 'shell = True' but not otherwise. Luca Cerone <luca.cerone@gmail.com> - 2013-05-26 16:58 -0700
      RE: Piping processes works with 'shell = True' but not otherwise. Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-27 03:14 +0300
        Re: Piping processes works with 'shell = True' but not otherwise. Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2013-05-29 19:39 +0200
          RE: Piping processes works with 'shell = True' but not otherwise. Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-29 22:31 +0300
          Re: Piping processes works with 'shell = True' but not otherwise. Cameron Simpson <cs@zip.com.au> - 2013-05-30 08:18 +1000
      Re: Piping processes works with 'shell = True' but not otherwise. Chris Angelico <rosuav@gmail.com> - 2013-05-27 18:28 +1000
        Re: Piping processes works with 'shell = True' but not otherwise. Luca Cerone <luca.cerone@gmail.com> - 2013-05-27 04:33 -0700
      Re: Piping processes works with 'shell = True' but not otherwise. Chris Rebert <clp2@rebertia.com> - 2013-05-29 10:17 -0700
        Re: Piping processes works with 'shell = True' but not otherwise. Luca Cerone <luca.cerone@gmail.com> - 2013-05-31 02:28 -0700
          Re: Piping processes works with 'shell = True' but not otherwise. Peter Otten <__peter__@web.de> - 2013-05-31 11:52 +0200

csiph-web