Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #10809 > unrolled thread

Snippet: The leanest Popen wrapper

Back to article view | Back to comp.lang.python

Contents

  Snippet: The leanest Popen wrapper Phlip <phlip2005@gmail.com> - 2011-08-03 08:29 -0700
    Re: Snippet: The leanest Popen wrapper Peter Otten <__peter__@web.de> - 2011-08-03 18:21 +0200
    Re: Snippet: The leanest Popen wrapper Thomas Jollans <t@jollybox.de> - 2011-08-03 18:39 +0200
    Re: Snippet: The leanest Popen wrapper Chris Rebert <clp2@rebertia.com> - 2011-08-03 10:27 -0700
      Re: Snippet: The leanest Popen wrapper Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2011-08-04 10:10 +0200
        Re: Snippet: The leanest Popen wrapper Chris Rebert <clp2@rebertia.com> - 2011-08-04 01:42 -0700
          Re: Snippet: The leanest Popen wrapper Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2011-08-04 11:25 +0200
    Re: Snippet: The leanest Popen wrapper Phlip <phlip2005@gmail.com> - 2011-08-03 11:04 -0700
    Re: Snippet: The leanest Popen wrapper Terry Reedy <tjreedy@udel.edu> - 2011-08-03 16:20 -0400
    Re: Snippet: The leanest Popen wrapper Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2011-08-04 10:24 +0200
    Re: Snippet: The leanest Popen wrapper Yves-Gwenael Bourhis <ybourhis@distroiwork4.com> - 2011-08-04 15:19 +0200

#10809 — Snippet: The leanest Popen wrapper

Groupies:

This is either a code snippet, if you like it, or a request for a
critique, if you don't.

I want to call a command and then treat the communication with that
command as an object. And I want to do it as application-specifically
as possible. Anyone could think of a way to productize this:

def command(*cmdz):

    process = Popen( flatten(cmdz),
                     shell= True,
                     stdout= subprocess.PIPE,
                     stderr= subprocess.PIPE,
                     bufsize= 4096 )

    def line():
        return process.stdout.readline().rstrip()

    def s():
        while True:
            l = line()
            if not l:  break
            yield l

    line.s = s

    return line

That leads to some syntactic sugar. For example, one truly demented
way to stream in an entire block and then treat it as one big string
is this:

print '\n'.join(command('ls').s())

The point of the command() complex is the ability to start a long
command and then fetch out individual lines from it:

line = command('find', '../..')

print 'lines'
print line()
print line()
print line()

print 'all'
print list(line.s())

If you need different pipe abilities, such as stdin, you can trivially
add them to the contents of command() (it's not productized on
purpose).

So I can take the line() functor and, for example, pin it to a View
object, or put it in another thread now, right?

--
  Phlip
  http://bit.ly/ZeekLand

[toc] | [next] | [standalone]

#10812

Phlip wrote:

> Groupies:

I smell a slight misperception of the audience you are addressing ;)

> This is either a code snippet, if you like it, or a request for a
> critique, if you don't.
> 
> I want to call a command and then treat the communication with that
> command as an object. And I want to do it as application-specifically
> as possible. Anyone could think of a way to productize this:
> 
> def command(*cmdz):
> 
>     process = Popen( flatten(cmdz),
>                      shell= True,
>                      stdout= subprocess.PIPE,
>                      stderr= subprocess.PIPE,
>                      bufsize= 4096 )

Protect your environment, don't let stderr pollute the nearby river ;)

>     def line():
>         return process.stdout.readline().rstrip()
> 
>     def s():
>         while True:
>             l = line()

At that point l may be empty because you have read the output completely or 
because there was an empty line that you rstripped to look like the end of 
file.

>             if not l:  break
>             yield l
> 
>     line.s = s
> 
>     return line

I think you are overdoing that closure/function factory thing a bit...

Seriously, you should reread the subprocess documentation to learn how to 
avoid deadlocks.

[toc] | [prev] | [next] | [standalone]

#10814

On 03/08/11 17:29, Phlip wrote:
> Groupies:
> 
> This is either a code snippet, if you like it, or a request for a
> critique, if you don't.
> 
> I want to call a command and then treat the communication with that
> command as an object. And I want to do it as application-specifically
> as possible. Anyone could think of a way to productize this:
> 
> def command(*cmdz):
> 
>     process = Popen( flatten(cmdz),
>                      shell= True,
>                      stdout= subprocess.PIPE,
>                      stderr= subprocess.PIPE,
>                      bufsize= 4096 )
> 
>     def line():
>         return process.stdout.readline().rstrip()
> 
>     def s():
>         while True:
>             l = line()
>             if not l:  break

This will ignore everything after a blank line. Intended?
It may be better not to use readline(), but to use the fact that it's an
iterable, and use next(process.stdout) to get each line. (and deal with
StopIteration accordingly -- or not)

>             yield l
> 
>     line.s = s
> 
>     return line
> 
> That leads to some syntactic sugar. For example, one truly demented
> way to stream in an entire block and then treat it as one big string
> is this:
> 
> print '\n'.join(command('ls').s())
> 
> The point of the command() complex is the ability to start a long
> command and then fetch out individual lines from it:
> 
> line = command('find', '../..')
> 
> print 'lines'
> print line()
> print line()
> print line()
> 
> print 'all'
> print list(line.s())
> 
> If you need different pipe abilities, such as stdin, you can trivially
> add them to the contents of command() (it's not productized on
> purpose).
> 
> So I can take the line() functor and, for example, pin it to a View
> object, or put it in another thread now, right?
> 
> --
>   Phlip
>   http://bit.ly/ZeekLand

[toc] | [prev] | [next] | [standalone]

#10818

On Wed, Aug 3, 2011 at 8:29 AM, Phlip <phlip2005@gmail.com> wrote:
> Groupies:
>
> This is either a code snippet, if you like it, or a request for a
> critique, if you don't.
>
> I want to call a command and then treat the communication with that
> command as an object. And I want to do it as application-specifically
> as possible. Anyone could think of a way to productize this:
>
> def command(*cmdz):
>
>    process = Popen( flatten(cmdz),

flatten() being defined as...?

>                     shell= True,

I would strongly encourage you to avoid shell=True. You really don't
want to have to worry about doing proper shell escaping yourself.

>                     stdout= subprocess.PIPE,
>                     stderr= subprocess.PIPE,
>                     bufsize= 4096 )

Cheers,
Chris

[toc] | [prev] | [next] | [standalone]

#10848

Am 03.08.2011 19:27 schrieb Chris Rebert:

>>                      shell= True,
>
> I would strongly encourage you to avoid shell=True.

ACK, but not because it is hard, but because it is unnecessary and 
unelegant at this point.

> You really don't want to have to worry about doing proper shell escaping yourself.

That's nothing to really worry about - just doing

def shellquote(*strs):
         return " ".join([
                 "'"+st.replace("'","'\\''")+"'"
                 for st in strs
         ])

would do perfectly: shellquote('echo', "'", '"', " ", "\n")
If you emit a command line over ssh, for example, you don't have another 
simple choice.

There are only worries if there is a shell which better shouldn't be 
named like this. As you generally cannot know what ugly things the user 
of your program does, it is better to avoid the additional shell layer.

So generally agree to what you say, but it is not the proper shell 
escaping one should worry about (it is so simple that one cannot call it 
"worry"), but the type of shell one talks with.

Thomas

[toc] | [prev] | [next] | [standalone]

#10850

On Thu, Aug 4, 2011 at 1:10 AM, Thomas Rachel
<nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de>
wrote:
> Am 03.08.2011 19:27 schrieb Chris Rebert:
>
>>>                     shell= True,
>>
>> I would strongly encourage you to avoid shell=True.
>
> ACK, but not because it is hard, but because it is unnecessary and unelegant
> at this point.
>
>> You really don't want to have to worry about doing proper shell escaping
>> yourself.
>
> That's nothing to really worry about - just doing
>
> def shellquote(*strs):
>        return " ".join([
>                "'"+st.replace("'","'\\''")+"'"
>                for st in strs
>        ])
>
> would do perfectly: shellquote('echo', "'", '"', " ", "\n")

I was considering the more general case where one of the strings may
have come from user input. You then need to also escape
$looks_like_a_var, `some_command`, and way more other such stuff that
your simple function doesn't cover. Even if the user is trusted, not
escaping such things can still lead to bizarre unintended
output/results.

If the commands are completely static, then yes, I agree that lack of
necessity then becomes the main argument against shell=True.

Cheers,
Chris
--
http://rebertia.com

[toc] | [prev] | [next] | [standalone]

#10852

Am 04.08.2011 10:42 schrieb Chris Rebert:

> I was considering the more general case where one of the strings may
> have come from user input. You then need to also escape
> $looks_like_a_var, `some_command`, and way more other such stuff that
> your simple function doesn't cover.

Even these things are harmless when included in ''s.

$ echo '`rm -rf .`' '$RANDOM'
`rm -rf .` $RANDOM

Thomas

[toc] | [prev] | [next] | [standalone]

#10820

> flatten() being defined as...?

Python Cookbook recipe 4.6

def flatten(sequence):  #  CONSIDER:  Reconcile with utils...
    for item in sequence:
        if isinstance(item, (list, tuple)):
            for subitem in flatten(list(item)):
                yield subitem
        else:
            yield item

>>                     shell= True,

> I would strongly encourage you to avoid shell=True. You really don't
> want to have to worry about doing proper shell escaping yourself.

Tx for helping me avoid reading up on it. I just copied it in.

I keep getting "fab not found" etc. when running 'fab command' thru
it. So then I ditch to os.system().

The long-term solution would be 'bash', '-c', 'yack yack yack' if you
want truly shelly things!

-- 
  Phlip
  http://c2.com/cgi/wiki?ZeekLand

[toc] | [prev] | [next] | [standalone]

#10827

On 8/3/2011 11:29 AM, Phlip wrote:

> This is either a code snippet, if you like it, or a request for a
> critique, if you don't.

A learning exercise but pretty useless otherwise. As others pointed out, 
immediately stripping off \n is a bug relative to *your* function 
description. Also, you yourself then give an example of joining with \n. 
But that does not restore the final \n. The rest duplicates the 
iteration ability of the .stdout file object.

For repeated execution of code like
   process.stdout.readline()
you can once create a 'packetized' bound method object like so
   cmdline = process.stdout.readline
and then, without repeating the attribute lookups, repeatedly call
   cmdline()
.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#10849

Am 03.08.2011 17:29 schrieb Phlip:
> Groupies:
>
> This is either a code snippet, if you like it, or a request for a
> critique, if you don't.

Well, at first, I cannot see the real point about it...


> def command(*cmdz):
>
>      process = Popen( flatten(cmdz),
>                       shell= True,
>                       stdout= subprocess.PIPE,
>                       stderr= subprocess.PIPE,
>                       bufsize= 4096 )
>
>      def line():
>          return process.stdout.readline().rstrip()
>
>      def s():
>          while True:
>              l = line()
>              if not l:  break
>              yield l
>
>      line.s = s
>
>      return line

I find it quite ugly. You get a function object with an attached 
generator (which has a strange and non-verbose name and) which might 
stop too early due to an empty line. Plus, you have no real control over 
the communication; you have no access to stdin or stderr. The latter 
might produce a lock if the process writes out too much on stderr.

Plus, you have no access to the exit code of the program.

And you lose information about if the stream ends with any combination 
of whitespace.


> That leads to some syntactic sugar. For example, one truly demented
> way to stream in an entire block and then treat it as one big string
> is this:
>
> print '\n'.join(command('ls').s())

What would work as well via

print Popen( ['ls'], stdout= subprocess.PIPE).stdout.read()
or
print Popen( ['ls'], stdout= subprocess.PIPE).communicate()[0]


> The point of the command() complex is the ability to start a long
> command and then fetch out individual lines from it:
>
> line = command('find', '../..')

sp = Popen( ['find', '../..'], stdout= subprocess.PIPE).stdout.read()
line = sp.stdout.readline
# if you want so: line = lambda: sp.stdout.readline().rstrip() - which
# might lose information as well...

> print 'lines'
> print line()
> print line()
> print line()
>
> print 'all'
> print list(line.s())
print list(iter(line, ''))


Thomas

[toc] | [prev] | [next] | [standalone]

#10860

Phlip a écrit ce mercredi 3 août 2011 17:29 dans <51b2d157-3fea-4f8e-80b4-
e7142629eca8@s21g2000pre.googlegroups.com> :

I made this because subprocess.Popen was giving me headaches ;-) :
http://pypi.python.org/pypi/commandwrapper/0.1

I use it here:
http://svn.mandriva.com/cgi-
bin/viewvc.cgi/soft/lab/doc4/django_doc4/doc4/utils/commandwrapper.py?revision=272275&view=markup
And here:
http://svn.deskolo.org/deskolo/browser/trunk/src/deskolo/util/commandwrapper.py

It's worth what it's worth, but I find it less headache_giving than 
subprocess.Popen is (reason I made it)

It's on Pypi so you can:
easy_install commandwrapper


usage example1:
===============
Ls = WrapCommand( 'ls -l')
GrepPdf = WrapCommand( 'grep pdf')
Wc = WrapCommand( 'wc -l')
Wc(GrepPdf(Ls)) # <- gives the result of "ls -l | grep pdf | wc -l"

usage example2:
===============
Ls = WrapCommand( 'ls -l | grep pdf | wc -l', shell=True)
Ls()


or imagine that the above commands may take a lot of time (and in parallel 
you want to do something else): 

usage example3:
===============
Ls = WrapCommand( 'ls -l')
GrepPdf = WrapCommand( 'grep pdf')
Wc = WrapCommand( 'wc -l')
Wc.stdin = GrepPdf
GrepPdf.stdin = Ls
Wc.start( )
#Do stuff
...
#finished doing stuff and wait for the result of the commands we launched
Wc.join()
Wc.results


I have put a lot of docstring in it as guidelines:

Code to Latest version:
=======================

#! /usr/bin/env python
# -*- coding: utf-8 -*-
#
## This file may be used under the terms of the GNU General Public
## License version 2.0 as published by the Free Software Foundation
## and appearing in the file LICENSE included in the packaging of
## this file.
##
## This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE
## WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
##

import shlex, subprocess
from threading import Thread
#from multiprocessing import Process as Thread, Event
from time import sleep

class WrapCommand (Thread):
    r"""Author : Yves-Gwenael Bourhis
    
    ==================================================
    Wrap a shell comand into a python threaded object.
    ==================================================
    
    Usage:
    ======
    
    You want to launch the following bash commands in a thread::
    
        [user@localhost ~]$ ls -l | grep pdf | wc -l
        5
    
    here is how you can do it::
    
        >>> Ls = WrapCommand( 'ls -l')
        >>> GrepPdf = WrapCommand( 'grep pdf')
        >>> Wc = WrapCommand( 'wc -l')
        >>> Wc.stdin = GrepPdf
        >>> GrepPdf.stdin = Ls
        >>> Wc.start( )
        >>> #Do stuff
        ...
        >>> Wc.join()
        >>> Wc.results
        ('5\n', '')
    
    the 'results' property is a tuple  (stdoutdata, stderrdata)
    
    You can also do it this way::
    
        >>> Ls = WrapCommand( 'ls -l | grep pdf | wc -l', shell=True)
        >>> Ls.start()
        >>> #Do stuff
        >>> Ls.join()
        >>> Ls.results[0]
        '5\n'
    
    You would need to specify 'shell=True' when the command
    you wish to execute is actually built into the shell.
    i.e.: on Windows if you use built in commands such as 'dir' or 'copy':
    http://docs.python.org/library/subprocess.html#subprocess.Popen
    
    The purpose of doing it in a thread is when the above commands may
    take a few hours, and that you want to perform other tasks in the
    meanwhile.
    You can check the process is still running with::
    
        >>> Wc.is_alive( )
        False
    
    'True' would be returned if still running.
    To terminate it prematurely (i.e. it deadlocked) you have the
    'terminate()', 'kill()' or 'send_signal(signal) methods which are
    self speaking.
    When you want to wait for the thread to end, use the 'join()' method:
    http://docs.python.org/library/threading.html#threading.Thread.join
    
    
    You want to launch the following bash commands without threading::
    
        [user@localhost ~]$ ls -l | grep pdf | wc -l
        5
    
    here is how you can do it::
    
        >>> Ls = WrapCommand( 'ls -l')
        >>> GrepPdf = WrapCommand( 'grep pdf')
        >>> Wc = WrapCommand( 'wc -l')
        >>> Wc(GrepPdf(Ls))
        '5\n'
    
    Avoid doing this for processes where a large amount of data is piped
    between each command.
    
    instead, do it this way::
    
        >>> Ls = WrapCommand( 'ls -l | grep pdf | wc -l', shell=True)
        >>> Ls()
        '5\n'
    
    Prefer the threaded method instead if this may take a long time and
    that you want to perform other tasks in the meanwhile.
    
    
    You can specify another shell for running commands::
    
        >>> Ls = WrapCommand( 'ls', shell=True, 
executable='C:/windows/System32/WindowsPowerShell/v1.0/powershell.exe')
        >>> print Ls()
    
            Directory : C:\Users\Yves\python_tests
    
        Mode                LastWriteTime     Length Name
        ----                -------------     ------ ----
        -a---        27/01/2011     00:14       7006 commandwrapper.py
        -a---        27/01/2011     00:15       7048 commandwrapper.pyc
    
    
    You can also use Context Management (with_item):
    http://docs.python.org/reference/compound_stmts.html#grammar-token-
with_item
    
    example::
    
        >>> with WrapCommand( 'ls -l') as Ls:
        ...     with WrapCommand( 'grep pdf') as GrepPdf:
        ...         with WrapCommand( 'wc -l') as Wc:
        ...             Wc.stdin = GrepPdf
        ...             GrepPdf.stdin = Ls
        ...             Wc.start( )
        ...             #Do stuff
        ...             Wc.join()
        ...
        >>> Wc.results
        ('5\n', '')
    
    You may also simply want to have a subprocess objet::
        
        >>> ls = WrapCommand( 'ls -l')
        >>> lscmd = ls.makeCmd()
        >>>
    
    the returned object (`lscmd` in the example above) is a standard 
subprocess.Popen object
    """
    
    pid = None
    returncode = None
    results = [None, None]
    exc_type = None
    exc_value = None
    traceback = None
    sent_signal = None
    dont_auto_communicate = False

    stdin = None
    stdout = subprocess.PIPE
    stderr = subprocess.PIPE

    shell = False
    executable = None
    command = None
    commands = None
    
    def __init__(self, command, shell=False, executable=None,
                 stdin=None, stdout=subprocess.PIPE, 
stderr=subprocess.PIPE):
        self.__parent = super(WrapCommand, self)
        self.cmd = None
        self.shell = shell
        self.executable = executable
        self.command = command
        self.commands = shlex.split(command)
        name = self.commands[0]
        self.stdin = stdin
        self.stdout = stdout
        self.stderr = stderr
        self.__parent.__init__(name=name)

    def prepareToRun(self, dont_auto_communicate=False):
        """
        returns the subprocess object::

            >>> Ls = WrapCommand( 'ls -l')
            >>> cmd = Ls.prepareToRun( )
            >>> result = cmd.communicate( )

        'cmd' is a subprocess.Popen object
        See http://docs.python.org/library/subprocess.html#subprocess.Popen

        'result' is a tuple (stdoutdata, stderrdata).
        See 
http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate
        """
        self.dont_auto_communicate = dont_auto_communicate
        if isinstance(self.stdin, str):
            stdin = subprocess.PIPE
        elif isinstance(self.stdin, self.__class__):
            self.stdin.prepareToRun(True)
            stdin = self.stdin.cmd.stdout
        else:
            stdin = self.stdin
        if self.shell:
            command = self.command
        else:
            command = self.commands
        self.cmd = subprocess.Popen(command,
                                    stdin=stdin,
                                    stdout=self.stdout,
                                    stderr=self.stderr,
                                    shell=self.shell,
                                    executable=self.executable)
        self.pid = self.cmd.pid
        return self.cmd
        
    makeCmd = prepareToRun
    make_cmd = prepareToRun

    def __enter__(self):
        return self

    def __call__(self, stdin=None):
        self.stdin = stdin
        self.run()
        return self.results[0]

    def run(self):
        self.prepareToRun()
        if not self.dont_auto_communicate:
            self.results = self.cmd.communicate(self.stdin)
        self.returncode = self.cmd.returncode

    def send_signal(self, signal):
        if self.is_alive() and self.cmd is not None:
            self.cmd.send_signal(signal)
            self.sent_signal = signal

    def terminate(self):
        if self.is_alive() and self.cmd is not None:
            self.cmd.terminate()
            self.sent_signal = 'SIGTERM'
        if hasattr(self.__parent, 'terminate'):
            self.__parent.terminate()

    def kill(self):
        if self.is_alive() and self.cmd is not None:
            self.cmd.kill()
            self.sent_signal = 'SIGKILL'

    def stop(self):
        if self.is_alive()and self.cmd is not None:
            self.terminate()
            if self.is_alive() and self.cmd is not None:
                sleep(1)
            if self.is_alive() and self.cmd is not None:
                self.kill()

    def __exit__(self, exc_type=None, exc_value=None, traceback=None):
        if self.is_alive() and self.cmd is not None:
            self.stop()
        self.exc_type = exc_type
        self.exc_value = exc_value
        self.traceback = traceback

    def __del__(self):
        self.__exit__()
        if self.cmd is not None:
            self.cmd.wait()
            del self.cmd

class WrapOnceCommand(WrapCommand):
    """
    Same as WrapCommand, but the cmd attribute which is a subprocess.Popen 
object will be created once and for all
    Therefore the run methode (or the object) can only be called once.
    The goal it to launch a command in a thread, and to have this command 
easily start/stopped from elsewhere.
    """
    def run(self):
        if self.cmd is None:    
            self.prepareToRun()
        if not self.dont_auto_communicate:
            self.results = self.cmd.communicate(self.stdin)
        self.returncode = self.cmd.returncode


Ok, it's not perfect yet, but makes subprocess.Popen easyier

-- 
Yves-Gwenael Bourhis

To contact me, replace "distroiwork4" by "mandriva" in my email.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web