Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #6481 > unrolled thread

How to catch a line with Popen

Back to article view | Back to comp.lang.python

Contents

  How to catch a line with Popen TheSaint <nobody@nowhere.net.no> - 2011-05-29 00:01 +0800
    Re: How to catch a line with Popen Nobody <nobody@nowhere.com> - 2011-05-28 23:15 +0100
    Re: How to catch a line with Popen Tim Roberts <timr@probo.com> - 2011-05-28 16:32 -0700
      Re: How to catch a line with Popen TheSaint <nobody@nowhere.net.no> - 2011-05-29 18:02 +0800
        Re: How to catch a line with Popen Chris Rebert <clp2@rebertia.com> - 2011-05-29 03:26 -0700
          Re: How to catch a line with Popen TheSaint <nobody@nowhere.net.no> - 2011-05-29 21:52 +0800
            Re: How to catch a line with Popen TheSaint <nobody@nowhere.net.no> - 2011-05-29 22:13 +0800
            Re: How to catch a line with Popen Chris Torek <nospam@torek.net> - 2011-05-30 01:02 +0000
              Re: How to catch a line with Popen TheSaint <nobody@nowhere.net.no> - 2011-05-30 23:29 +0800
                Re: How to catch a line with Popen Chris Torek <nospam@torek.net> - 2011-05-30 21:33 +0000
                  Re: How to catch a line with Popen TheSaint <nobody@nowhere.net.no> - 2011-05-31 19:33 +0800

#6481 — How to catch a line with Popen

Hello.
I'm looking into subprocess.Popen docs.
I've launch the program with its arguments and that's smooth. I'm expecting 
to read the output by *comunicate()* at every line that prgram may blow 
during the process, but the output is given only when the child process is 
ended.
I'd like to process the lines to display an information in percentage during 
the running time of the child. Sorry but I'm poor of know-how, so I'm stuck 
over to find a clue.

-- 
goto /dev/null

[toc] | [next] | [standalone]

#6493

On Sun, 29 May 2011 00:01:56 +0800, TheSaint wrote:

> I'm looking into subprocess.Popen docs. I've launch the program with its
> arguments and that's smooth. I'm expecting to read the output by
> *comunicate()* at every line that prgram may blow during the process, but
> the output is given only when the child process is ended.

.communicate() reads until EOF, .wait()s for the process to terminate,
then returns the output as a pair of strings.

If you want to read data while the process is running, use the process'
.stdout and/or .stderr members, e.g.:

	p = subprocess.Popen(...)
	for line in p.stdout:
	    ...
	p.wait()

Don't forget to .wait() on the process when you're done, regardless of
whether you actually need the exit code.

[toc] | [prev] | [next] | [standalone]

#6497

TheSaint <nobody@nowhere.net.no> wrote:
>
>I'm looking into subprocess.Popen docs.
>I've launch the program with its arguments and that's smooth. I'm expecting 
>to read the output by *comunicate()* at every line that prgram may blow 
>during the process, but the output is given only when the child process is 
>ended.
>I'd like to process the lines to display an information in percentage during 
>the running time of the child.

Are you specifying a buffer size in the Popen command?  If not, then the
Python side of things is unbuffered.  It's possible that the application
you are launching is buffering its output, so that it doesn't SEND until
the application ends.
-- 
Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.

[toc] | [prev] | [next] | [standalone]

#6524

Tim Roberts wrote:

> Are you specifying a buffer size in the Popen command?  If not, then the
> Python side of things is unbuffered

The buffer is as per default. The program reports one line around 1/2 second 
time.
I think I'll look into the option as Nobody states:

        p = subprocess.Popen(...)
        for line in p.stdout:
            ...
        p.wait()

It is strange that would take a for cycle, rather than catching the line on-
the-fly. I can judge it now, I'm going to try it out.

-- 
goto /dev/null

[toc] | [prev] | [next] | [standalone]

#6525

On Sun, May 29, 2011 at 3:02 AM, TheSaint <nobody@nowhere.net.no> wrote:
> Tim Roberts wrote:
>
>> Are you specifying a buffer size in the Popen command?  If not, then the
>> Python side of things is unbuffered
>
> The buffer is as per default. The program reports one line around 1/2 second
> time.
> I think I'll look into the option as Nobody states:
>
>        p = subprocess.Popen(...)
>        for line in p.stdout:
>            ...
>        p.wait()
>
> It is strange that would take a for cycle, rather than catching the line on-
> the-fly. I can judge it now, I'm going to try it out.

What do you mean by "on-the-fly" in this context?

Cheers,
Chris

[toc] | [prev] | [next] | [standalone]

#6537

Chris Rebert wrote:

> What do you mean by "on-the-fly" in this context

I just suppose to elaborate the latest line, as soon it's written on the 
pipe, and print some result on the screen.
Imaging something like

 p= Popen(['ping','-c40','www.google.com'], stdout=PIPE)
 for line in p.stdout:
     print(str(line).split()[7])

I'd like to see something like *time=54.4*
This is just an example, where if we remove the "-c40" on the command line, 
I'd expect to read the latest line(s), until the program will be killed.

-- 
goto /dev/null

[toc] | [prev] | [next] | [standalone]

#6540

TheSaint wrote:

> I just suppose to elaborate the latest line, as soon it's written on the
> pipe, and print some result on the screen.

I think some info is also here:
http://alexandredeverteuil.blogspot.com/
-- 
goto /dev/null

[toc] | [prev] | [next] | [standalone]

#6602

In article <irtj2o$h0m$1@speranza.aioe.org>
TheSaint  <nobody@nowhere.net.no> wrote:
>Chris Rebert wrote:
>I just suppose to elaborate the latest line, as soon it's written on the 
>pipe, and print some result on the screen.
>Imaging something like
>
> p= Popen(['ping','-c40','www.google.com'], stdout=PIPE)
> for line in p.stdout:
>     print(str(line).split()[7])
>
>I'd like to see something like *time=54.4*
>This is just an example, where if we remove the "-c40" on the command line, 
>I'd expect to read the latest line(s), until the program will be killed.

In at least some versions of Python 2, file-like object "next"
iterators do not "work right" with unbuffered (or line-buffered)
pipe-file-objects.  (This may or may not be fixed in Python 3.)

A simple workaround is a little generator using readline():

def line_at_a_time(fileobj):
    """
    Return one line at a time from a file-like object.
    Works around the iter behavior of pipe files in
    Python 2.x, e.g., instead of "for line in file" you can
    write "for line in line_at_a_time(file)".
    """
    while True:
        line = fileobj.readline()
        if not line:
            return
        yield line

Adding this to your sample code gives something that works for me,
provided I fiddle with it to make sure that the only lines
examined are those with actual ping times:

    p = subprocess.Popen(["ping", "-c5", "www.google.com"],
        stdout = subprocess.PIPE)
    for lineno, line in enumerate(line_at_a_time(p.stdout)):
        if 1 <= lineno <= 5:
            print line.split()[6]
        else:
            print line.rstrip('\n')
    p.wait() # discard final result

(Presumably the enumerate() trick would not be needed in whatever
you really use.)
-- 
In-Real-Life: Chris Torek, Wind River Systems
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W)  +1 801 277 2603
email: gmail (figure it out)      http://web.torek.net/torek/index.html

[toc] | [prev] | [next] | [standalone]

#6665

Chris Torek wrote:

> In at least some versions of Python 2

I'm with P3k :P. However thank you for your guidelines.
Last my attempt was to use a *for* p.wait() , as mentioned earlier

That looks good enough. I noted some little delay for the first lines, 
mostly sure Popen assign some buffer even it is not set.

Haven't you try a perpetual ping, how would be the line_at_a_time ?

-- 
goto /dev/null

[toc] | [prev] | [next] | [standalone]

#6673

>Chris Torek wrote:
>> In at least some versions of Python 2
[the "file"-type object iterators behave badly with pipes]

(This may still be true in Python 3, I just have no experience with
Py3k.  "At least some version of Python 2" means "the ones I have
access to, and have tried." :-) )

In article <is0d44$d7m$1@speranza.aioe.org>
TheSaint  <nobody@nowhere.net.no> wrote:
>I'm with P3k :P. However thank you for your guidelines.
>Last my attempt was to use a *for* p.wait() , as mentioned earlier

If you have a process that has not yet terminated and that you
must stop from your own python program, calling the wait() method
will wait forever (because you are now waiting for yourself, in
effect -- waiting for yourself to terminate the other process).

The only time to call p.wait() (or p.communicate(), which itself
calls the wait() method) is when you believe the subprocess is
on its wao to terminating -- in this case, after you force it
to do so.

>That looks good enough. I noted some little delay for the first lines, 
>mostly sure Popen assign some buffer even it is not set.

According to the documentation, the default buffer size of Python 2
is 0, which is passed to fdopen() and makes the resulting files
unbuffered.  I recall some sort of changes discussed for Py3k though.

>Haven't you try a perpetual ping, how would be the line_at_a_time ?

Since it is a generator that only requests another line when called,
it should be fine.  (Compare to the itertools "cycle" and "repeat"
generators, for instance, which "return" an infinite sequence.)
-- 
In-Real-Life: Chris Torek, Wind River Systems
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W)  +1 801 277 2603
email: gmail (figure it out)      http://web.torek.net/torek/index.html

[toc] | [prev] | [next] | [standalone]

#6725

Chris Torek wrote:

> Since it is a generator that only requests another line when called,
> it should be fine
Is it, then, that until the new itaration, the callee is on pause?

-- 
goto /dev/null

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web