Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #12790 > unrolled thread

One line command line filter

Back to article view | Back to comp.lang.python

Contents

  One line command line filter Jon Redgrave <jredgrave@capisco.com> - 2011-09-05 13:38 -0700
    Re: One line command line filter Thomas Jollans <t@jollybox.de> - 2011-09-05 22:47 +0200
      Re: One line command line filter Jon Redgrave <jredgrave@capisco.com> - 2011-09-05 14:32 -0700
        Re: One line command line filter Terry Reedy <tjreedy@udel.edu> - 2011-09-05 19:02 -0400
    Re: One line command line filter Terry Reedy <tjreedy@udel.edu> - 2011-09-05 18:21 -0400
      Re: One line command line filter Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-09-06 09:18 +1000
        Re: One line command line filter Terry Reedy <tjreedy@udel.edu> - 2011-09-05 21:25 -0400
        Re: One line command line filter Hans Mulder <hansmu@xs4all.nl> - 2011-09-06 13:15 +0200
    Re: One line command line filter Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-09-06 09:29 +1000
    Re: One line command line filter John Wiegley <jwiegley@gmail.com> - 2011-09-06 01:51 -0500
    Re: One line command line filter Peter Otten <__peter__@web.de> - 2011-09-06 09:53 +0200

#12790 — One line command line filter

It seems unreasonably hard to write simple one-line unix command line
filters in python:

eg: ls | python -c "<something>  print x.upper()"

to get at sys.stdin or similar needs an import, which makes a
subsequent for-loop illegal.
python -c "import sys; for x in sys.stdin(): print x" <<- SyntaxError

Am I missing something obvious?

The best I've come up with is to use sitecustomize.py to add to
__builtin__
def stdin():
  import sys
  for x in sys.stdin():
    if not x: return
    yield x.strip()
import __builtin__
__builtin__.stdin = stdin

This allows
ls | python -c "for x in stdin(): print x.upper()"

Is there a better solution - if not is this worth a PEP?

[toc] | [next] | [standalone]

#12791

On 05/09/11 22:38, Jon Redgrave wrote:
> It seems unreasonably hard to write simple one-line unix command line
> filters in python:
> 
> eg: ls | python -c "<something>  print x.upper()"
> 
> to get at sys.stdin or similar needs an import, which makes a
> subsequent for-loop illegal.
> python -c "import sys; for x in sys.stdin(): print x" <<- SyntaxError
> 
> Am I missing something obvious?

ls | python -c "for line in __import__('sys').stdin: print (line.upper())"

[toc] | [prev] | [next] | [standalone]

#12792

> > Am I missing something obvious?
>
> ls | python -c "for line in __import__('sys').stdin: print (line.upper())"

Ah, so I am missing something - it is possible - but 'obvious'?
Do people think it should be more accessible

[toc] | [prev] | [next] | [standalone]

#12796

On 9/5/2011 5:32 PM, Jon Redgrave wrote:
>>> Am I missing something obvious?
>>
>> ls | python -c "for line in __import__('sys').stdin: print (line.upper())"
>
> Ah, so I am missing something - it is possible - but 'obvious'?
> Do people think it should be more accessible

__import__ is well-documented and is listed in the index of the builtin 
functions chapter. "Direct use of __import__() is rare, except in cases 
where you want to import a module whose name is only known at runtime." 
could be explanded to include "or where you want to import as part of an 
expression"

Every Python programmer should peruse that chapter to learn what is 
available for possible future use.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#12794

On 9/5/2011 4:38 PM, Jon Redgrave wrote:
> It seems unreasonably hard to write simple one-line unix command line
> filters in python:
>
> eg: ls | python -c "<something>   print x.upper()"
>
> to get at sys.stdin or similar needs an import, which makes a
> subsequent for-loop illegal.
> python -c "import sys; for x in sys.stdin(): print x"<<- SyntaxError
>
> Am I missing something obvious?

The doc says "-c <command>
Execute the Python code in command. command can be one or more 
statements separated by newlines,"

However, I have no idea how to put newlines into a command-line string. 
Changing '; ' to '\n' does not work, at least not in Windows. The '\n' 
is passed on to Python as 2 chars, and the '\' is seen as the 
line-continuation char and the 'n' as illegal following it. Will a *nix 
shell 'cook' the string and convert '\n' to a literal newline before 
passing it to Python?

For *nix, I would expect the <<EOF mechanism to work. Have you tried that?

That said, Python is designed for named multiple-statement programs,
just as it is designed for named multiple-statement functions. Or it is 
designed for interactive work.

The following works:
dir | python -c "while True: print(input())"

except that it finishes with an error traceback:
Traceback (most recent call last):
   File "<string>", line 1, in <module>
EOFError: EOF when reading a line

Perhaps tolerable if Python is at the end of the pipe, not otherwise.

 > Is there a better solution - if not is this worth a PEP?

PEPs have to propose a concrete solution, preferably with some previous 
discussion. I take is that your proposal would be to add another builtin 
means to access stdin.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#12799

Terry Reedy wrote:

> The doc says "-c <command>
> Execute the Python code in command. command can be one or more
> statements separated by newlines,"
> 
> However, I have no idea how to put newlines into a command-line string.

I imagine that it depends on the shell you are using, but bash on Linux
makes it simple: double quotes "..." are like Python's triple-quoted
strings in that they can include newlines.

[steve@sylar python]$ ls f*.py | python -c "import sys
> print sys.stdin.read()"
factorial.py
fetchqm.py
fib.py
fileutils.py
findsingle.py
fixascii.py
fix.py
frange.py
frequencies.py


Other shells may be different.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#12805

On 9/5/2011 7:18 PM, Steven D'Aprano wrote:
> Terry Reedy wrote:
>
>> The doc says "-c<command>
>> Execute the Python code in command. command can be one or more
>> statements separated by newlines,"
>>
>> However, I have no idea how to put newlines into a command-line string.
>
> I imagine that it depends on the shell you are using, but bash on Linux
> makes it simple: double quotes "..." are like Python's triple-quoted
> strings in that they can include newlines.
>
> [steve@sylar python]$ ls f*.py | python -c "import sys
>> print sys.stdin.read()"
> factorial.py
> fetchqm.py

I was guessing that whoever wrote the doc could do something like that. 
As far as I know, there is no way to escape a newline with Windows 
cmd.exe. (Someone please tell me if I am wrong!) An "unmatched" quote is 
either ignored or matched by the newline!

C:\Programs\Python32> python -c "print('haha')
haha

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#12827

On 6/09/11 01:18:37, Steven D'Aprano wrote:
> Terry Reedy wrote:
>
>> The doc says "-c<command>
>> Execute the Python code in command. command can be one or more
>> statements separated by newlines,"
>>
>> However, I have no idea how to put newlines into a command-line string.
>
> I imagine that it depends on the shell you are using, but bash on Linux
> makes it simple: double quotes "..." are like Python's triple-quoted
> strings in that they can include newlines.

Single quotes in bash are more similar to triple quotes in Python, in
that some shell syntax is recognized within double quoted string, but
not within single quoted strings:

$ python -c 'import sys
print `sys.version`'
'2.7.1 (r271:86882M, Nov 30 2010, 10:35:34) \n[GCC 4.2.1 (Apple Inc. 
build 5664)]'

$ python -c "import sys
 > print `sys.copyright`"
-bash: sys.copyright: command not found

When using single quotes, the `` are passed to Python, which interprets
them as a call to repr().  With double quotes, the shell interprets ``
as a sub-command.

> Other shells may be different.

Most *nix shells are similar to bash. The ones that are different
are csh and tcsh: they require a backslash before each newline:

$ tcsh
% ls f*.py | python -c 'import sys \
? print sys.stdin.read()'
filecmp.py
fileinput.py
fnmatch.py
formatter.py
fpformat.py
fractions.py
ftplib.py
functools.py
%

Sometimes you need two backslashes, one for csh and one for Python:

% python -c 'print "spam spam spam spam spam spam spam spam " \\
? "spam spam spam spam spam"'
spam spam spam spam spam spam spam spam spam spam spam spam spam
%

Hope this helps,

-- HansM

[toc] | [prev] | [next] | [standalone]

#12801

Jon Redgrave wrote:

> It seems unreasonably hard to write simple one-line unix command line
> filters in python:
> 
> eg: ls | python -c "<something>  print x.upper()"

Python is neither bash nor Perl. It is not intended to compete in the "quick
and dirty one-liner commands" stakes.

However, you might like to consider ipython, which is intended as a complete
Python shell, not just an interactive interpreter.

http://ipython.org/


[...]
> The best I've come up with is to use sitecustomize.py to add to
> __builtin__

If you're a system administrator who wants to replace bash one-liners at the
command line with Python one-liners, sure, why not?

If you build up a useful collection of sys admin tools or recipes, you might
like to consider sharing them. Perhaps if Python was used more by sys
admins, there might be less resistance to making it easier to compete with
bash one-liners.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#12815

>>>>> Jon Redgrave <jredgrave@capisco.com> writes:

> It seems unreasonably hard to write simple one-line unix command line
> filters in python:

> eg: ls | python -c "<something> print x.upper()"

[...]

> Is there a better solution - if not is this worth a PEP?

Have you looked at PyP (http://code.google.com/p/pyp)?

John

[toc] | [prev] | [next] | [standalone]

#12817

Jon Redgrave wrote:

> It seems unreasonably hard to write simple one-line unix command line
> filters in python:
> 
> eg: ls | python -c "<something>  print x.upper()"
> 
> to get at sys.stdin or similar needs an import, which makes a
> subsequent for-loop illegal.
> python -c "import sys; for x in sys.stdin(): print x" <<- SyntaxError
> 
> Am I missing something obvious?
> 
> The best I've come up with is to use sitecustomize.py to add to
> __builtin__
> def stdin():
>   import sys
>   for x in sys.stdin():
>     if not x: return
>     yield x.strip()
> import __builtin__
> __builtin__.stdin = stdin
> 
> This allows
> ls | python -c "for x in stdin(): print x.upper()"
> 
> Is there a better solution - if not is this worth a PEP?

$ touch alpha beta gamma omega
$ ls | python -c 'import sys; sys.stdout.writelines(s.upper() for s in 
sys.stdin if s.startswith(("a", "o")))'
ALPHA
OMEGA

If you are doing this a lot, why don't you write a helper script and invoke 
that? 

$ ls | pyfilter.py -f '"m" in line' -s 'lines = (line + line[::-1] for line 
in map(str.strip, lines))' -s'import re' -p 're.compile(r"(([a-
z])\2)").sub(lambda m: m.group(1).upper(), line)'
alphAAhpla
betAAteb
gaMMAAMMag
omegAAgemo

This relies on the convention that a single line of input is accessible as 
"line" and the complete input is called "lines". Of course the same can be 
done with python -c ..., -- and it is even more readable:

$ ls | python -c 'import re, sys
for line in sys.stdin:
>     line = line.strip()
>     line = line + line[::-1]
>     print re.compile(r"(([a-z])\2)").sub(lambda m: m.group(1).upper(), 
line)
> '
alphAAhpla
betAAteb
gaMMAAMMag
omegAAgemo

> Is there a better solution - if not is this worth a PEP?

The python interpreter could accept multiple -c arguments, but to see how 
this will be received a post on python-ideas should be sufficient.

For the sake of completeness here's the script I used to produce the example 
above:

$ cat pyfilter.py
#!/usr/bin/env python
import sys

def main():
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "-s", "--setup",
        action="append", default=[],
        dest="setups", metavar="SETUP")
    parser.add_argument(
        "-f", "--filter",
        action="append", default=[],
        dest="filters", metavar="FILTER")
    parser.add_argument("-p", "--print", dest="format")

    args = parser.parse_args()
    lines = sys.stdin
    for setup in args.setups:
        exec setup
    for line in lines:
        line = line.rstrip("\n")
        for filter in args.filters:
            if not eval(filter):
                continue
        if args.format:
            line = eval(args.format)
        try:
            print line
        except IOError as e:
            if e.errno == 32: # broken pipe
                break
            raise
if __name__ == "__main__":
    main()

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web