Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #97313 > unrolled thread

Only getting the first 6 lines

Back to article view | Back to comp.lang.python

Contents

  Only getting the first 6 lines Cecil Westerhof <Cecil@decebal.nl> - 2015-10-01 23:58 +0200
    Re: Only getting the first 6 lines Ian Kelly <ian.g.kelly@gmail.com> - 2015-10-01 16:58 -0600
    Re: Only getting the first 6 lines Cameron Simpson <cs@zip.com.au> - 2015-10-02 08:50 +1000
      Re: Only getting the first 6 lines Cecil Westerhof <Cecil@decebal.nl> - 2015-10-02 13:08 +0200
    Re: Only getting the first 6 lines Peter Otten <__peter__@web.de> - 2015-10-02 09:37 +0200
      Re: Only getting the first 6 lines Cecil Westerhof <Cecil@decebal.nl> - 2015-10-02 13:11 +0200

#97313 — Only getting the first 6 lines

I want to get the first 6 lines of ps output. For this I use:
========================================================================
from subprocess import check_output

ps_command = ('ps', '-eo', 'user,pid,pcpu,pmem,stat,start,time,cmd', '--sort')
message = '\n'.join(check_output(ps_command + ('-%cpu',)).decode("utf-8").splitlines()[0:6])
========================================================================

It works, but does not look very efficient. Is there a better way to
do this?

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [next] | [standalone]

#97316

On Thu, Oct 1, 2015 at 3:58 PM, Cecil Westerhof <Cecil@decebal.nl> wrote:
> I want to get the first 6 lines of ps output. For this I use:
> ========================================================================
> from subprocess import check_output
>
> ps_command = ('ps', '-eo', 'user,pid,pcpu,pmem,stat,start,time,cmd', '--sort')
> message = '\n'.join(check_output(ps_command + ('-%cpu',)).decode("utf-8").splitlines()[0:6])
> ========================================================================
>
> It works, but does not look very efficient. Is there a better way to
> do this?

Instead of using check_output which reads the entire output, you could
create a Popen with stdin=None, stdout=PIPE, stderr=None, and then
read just the first six lines. Don't forget to clean up the subprocess
afterward.

[toc] | [prev] | [next] | [standalone]

#97317

On 01Oct2015 23:58, Cecil Westerhof <Cecil@decebal.nl> wrote:
>I want to get the first 6 lines of ps output. For this I use:
>========================================================================
>from subprocess import check_output
>
>ps_command = ('ps', '-eo', 'user,pid,pcpu,pmem,stat,start,time,cmd', '--sort')
>message = '\n'.join(check_output(ps_command + ('-%cpu',)).decode("utf-8").splitlines()[0:6])
>========================================================================
>
>It works, but does not look very efficient. Is there a better way to
>do this?

It depends what you mean by inefficient. I'm presuming you mean that this:

  - sucks in all the output of ps instead of just the first 6 lines

  - sucks all the output into memory instead of just some of it

  - does more in-memory work in splitlines()

  - needs ps to run to completion instead of just long enough to print 6 lines

You could just read six lines of output from ps. Something like this 
(untested):


  lines = []
  for lineno, line in \
        enumerate(Popen(ps_command + ('-%cpu',),
                        stdin=NULL, stdout=PIPE).stdout,
                  1):
    lines.append(line.decode("utf-8"))
    if lineno >= 6:
      break

This works because in a miracle of foresight you can split binary data streams 
(the stdout) on line breaks, so you can fetch "binary" lines and decode them 
individually.

This approach is important if the output of your command is large or slow, as 
it does not need to suck the whole thing into memory or to wait for it to 
finish a long procedure. With "ps" these issues are pretty minor; with some 
other programs it can be significant, especially if the other program _doesn't_ 
terminate (consider "tail -f" of an active log file).

Cheers,
Cameron Simpson <cs@zip.com.au>

[toc] | [prev] | [next] | [standalone]

#97330

On Friday  2 Oct 2015 00:50 CEST, Cameron Simpson wrote:

> On 01Oct2015 23:58, Cecil Westerhof <Cecil@decebal.nl> wrote:
>> I want to get the first 6 lines of ps output. For this I use:
>> ========================================================================
>> from subprocess import check_output
>>
>> ps_command = ('ps', '-eo',
>> 'user,pid,pcpu,pmem,stat,start,time,cmd', '--sort') message =
>> '\n'.join(check_output(ps_command +
>> ('-%cpu',)).decode("utf-8").splitlines()[0:6])
>> ========================================================================
>>
>> It works, but does not look very efficient. Is there a better way
>> to do this?
>
> It depends what you mean by inefficient. I'm presuming you mean that
> this:
>
> - sucks in all the output of ps instead of just the first 6 lines
>
> - sucks all the output into memory instead of just some of it
>
> - does more in-memory work in splitlines()

Yes.


> - needs ps to run to completion instead of just long enough to print
>   6 lines

No, because of the sort, ps has to generate all lines.
>
> You could just read six lines of output from ps. Something like this
> (untested):
>
>
> lines = []
> for lineno, line in \
> enumerate(Popen(ps_command + ('-%cpu',),
> stdin=NULL, stdout=PIPE).stdout,
> 1):
> lines.append(line.decode("utf-8"))
> if lineno >= 6:
> break


I wrote the following general function for it:
========================================================================
def log_resource_usage(command, resource_type, nr_of_lines):
    lines   = []
    process = Popen(command, stdin = None, stdout = PIPE, stderr = None)
    for lineno, line in enumerate(process.stdout, 1):
        lines.append(line.decode('utf-8').strip())
        if lineno >= nr_of_lines:
            break
    message = '\n'.join(lines)
    cursor.execute(insert_message, (resource_type, message))
    process.wait()
========================================================================


> This approach is important if the output of your command is large or
> slow, as it does not need to suck the whole thing into memory or to
> wait for it to finish a long procedure. With "ps" these issues are
> pretty minor; with some other programs it can be significant,
> especially if the other program _doesn't_ terminate (consider "tail
> -f" of an active log file).

Well ps generates more as 400 lines. Not a real problem, but I like to
solve problems before they become a problem. ;-)

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [prev] | [next] | [standalone]

#97327

Cecil Westerhof wrote:

> I want to get the first 6 lines of ps output. For this I use:
> ========================================================================
> from subprocess import check_output
> 
> ps_command = ('ps', '-eo', 'user,pid,pcpu,pmem,stat,start,time,cmd',
> '--sort') message = '\n'.join(check_output(ps_command +
> ('-%cpu',)).decode("utf-8").splitlines()[0:6])
> ========================================================================
> 
> It works, but does not look very efficient. Is there a better way to
> do this?

Efficiency be damned, readability counts ;)

With that in mind here's a little code cleanup:

 
import subprocess

def text_head(text, n):
    return "\n".join(text.split("\n", n)[:n])

def ps(sort=None):
    cmd = ['ps', '-eo', 'user,pid,pcpu,pmem,stat,start,time,cmd']
    if sort is not None:
        cmd += ["--sort", sort]
    return subprocess.check_output(cmd, universal_newlines=True)

if __name__ == "__main__":
    print(text_head(ps("-%cpu"), 6))

For long strings and small n text.split(\n", n)[:n] is a bit faster than 
text.splitlines()[n]

$ wc -l fodder.txt
969 fodder.txt
$ python3 -m timeit -s 'text = open("fodder.txt").read()' 'text.split("\n", 
6)[:6]'
100000 loops, best of 3: 7.54 usec per loop
$ python3 -m timeit -s 'text = open("fodder.txt").read()' 'text.splitlines()
[:6]'
1000 loops, best of 3: 215 usec per loop

but the effect on the total time to run the code should be negligable.

[toc] | [prev] | [next] | [standalone]

#97331

On Friday  2 Oct 2015 09:37 CEST, Peter Otten wrote:

> Cecil Westerhof wrote:
>
>> I want to get the first 6 lines of ps output. For this I use:
>> ========================================================================
>> from subprocess import check_output
>>
>> ps_command = ('ps', '-eo',
>> 'user,pid,pcpu,pmem,stat,start,time,cmd', '--sort') message =
>> '\n'.join(check_output(ps_command +
>> ('-%cpu',)).decode("utf-8").splitlines()[0:6])
>> ========================================================================
>>
>> It works, but does not look very efficient. Is there a better way
>> to do this?
>
> Efficiency be damned, readability counts ;)

The second is more important as the first, but if you can have both …
:-)

I already made a function for it. See my other response.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web