Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #65210 > unrolled thread

piping with subprocess

Back to article view | Back to comp.lang.python

Contents

  piping with subprocess Rick Dooling <rpdooling@gmail.com> - 2014-02-01 04:19 -0800
    Re: piping with subprocess Peter Otten <__peter__@web.de> - 2014-02-01 13:54 +0100
      Re: piping with subprocess Rick Dooling <rpdooling@gmail.com> - 2014-02-01 05:54 -0800
        Re: piping with subprocess Rick Dooling <rpdooling@gmail.com> - 2014-02-01 06:00 -0800
          Re: piping with subprocess Rick Dooling <rpdooling@gmail.com> - 2014-02-01 07:35 -0800
            Re: piping with subprocess Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-02-01 15:59 +0000
        Re: piping with subprocess Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-02-01 14:28 +0000
        Re: piping with subprocess Peter Otten <__peter__@web.de> - 2014-02-01 18:32 +0100
    Re: piping with subprocess Daniel da Silva <var.mail.daniel@gmail.com> - 2014-02-01 07:40 -0500

#65210 — piping with subprocess

I spent half a day trying to convert this bash script (on Mac)

textutil -convert html $1 -stdout | pandoc -f html -t markdown -o $2

into Python using subprocess pipes.

It works if I save the above into a shell script called convert.sh and then do

subprocess.check_call(["convert.sh", file, markdown_file]) 

where file and markdown_file are variables.

But otherwise my piping attempts fail.

Could someone show me how to pipe in subprocess. Yes, I've read the doc, especially

http://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline

But I'm a feeble hobbyist, not a computer scientist.

Thanks

RD

[toc] | [next] | [standalone]

#65212

Rick Dooling wrote:

> I spent half a day trying to convert this bash script (on Mac)
> 
> textutil -convert html $1 -stdout | pandoc -f html -t markdown -o $2
> 
> into Python using subprocess pipes.
> 
> It works if I save the above into a shell script called convert.sh and
> then do
> 
> subprocess.check_call(["convert.sh", file, markdown_file])
> 
> where file and markdown_file are variables.
> 
> But otherwise my piping attempts fail.

It is always a good idea to post your "best effort" failed attempt, if only 
to give us an idea of your level of expertise.

> Could someone show me how to pipe in subprocess. Yes, I've read the doc,
> especially
> 
> http://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline
> 
> But I'm a feeble hobbyist, not a computer scientist.

Try to convert the example from the above page

"""
output=`dmesg | grep hda`
# becomes
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]
"""

to your usecase. Namely, replace

["dmesg"] --> ["textutil", "-convert", "html", infile, "-stdout"]
["grep", "hda"] --> ["pandoc", "-f", "html", "-t", "marktown", "-o",
                     outfile]

Don't forget to set

infile = ... 
outfile = ... 

to filenames (with absolute paths, to avoid one source of error).
If that doesn't work post the code you wrote along with the error messages.

[toc] | [prev] | [next] | [standalone]

#65214

On Saturday, February 1, 2014 6:54:09 AM UTC-6, Peter Otten wrote:
> Rick Dooling wrote:
> 
> 
> 
> > I spent half a day trying to convert this bash script (on Mac)
> 
> > 
> 
> > textutil -convert html $1 -stdout | pandoc -f html -t markdown -o $2
> 
> > 
> 
> > into Python using subprocess pipes.
> 
> > 
> 
> > It works if I save the above into a shell script called convert.sh and
> 
> > then do
> 
> > 
> 
> > subprocess.check_call(["convert.sh", file, markdown_file])
> 
> > 
> 
> > where file and markdown_file are variables.
> 
> > 
> 
> > But otherwise my piping attempts fail.
> 
> 
> 
> It is always a good idea to post your "best effort" failed attempt, if only 
> 
> to give us an idea of your level of expertise.
> 
> 
> 
> > Could someone show me how to pipe in subprocess. Yes, I've read the doc,
> 
> > especially
> 
> > 
> 
> > http://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline
> 
> > 
> 
> > But I'm a feeble hobbyist, not a computer scientist.
> 
> 
> 
> Try to convert the example from the above page
> 
> 
> 
> """
> 
> output=`dmesg | grep hda`
> 
> # becomes
> 
> p1 = Popen(["dmesg"], stdout=PIPE)
> 
> p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
> 
> p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
> 
> output = p2.communicate()[0]
> 
> """
> 
> 
> 
> to your usecase. Namely, replace
> 
> 
> 
> ["dmesg"] --> ["textutil", "-convert", "html", infile, "-stdout"]
> 
> ["grep", "hda"] --> ["pandoc", "-f", "html", "-t", "marktown", "-o",
> 
>                      outfile]
> 
> 
> 
> Don't forget to set
> 
> 
> 
> infile = ... 
> 
> outfile = ... 
> 
> 
> 
> to filenames (with absolute paths, to avoid one source of error).
> 
> If that doesn't work post the code you wrote along with the error messages.

p1 = subprocess.Popen(["textutil", "-convert", "html", file], stdout=subprocess.PIPE)
p2 = subprocess.check_call(["pandoc", "-f", "html", "-t", "markdown", "-o", markdown_file], stdin=p1.stdout, stdout=subprocess.PIPE)
p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]

Errors

Traceback (most recent call last):
  File "/Users/me/Python/any2pandoc.py", line 70, in <module>
    convert_word_file(file, markdown_file)
  File "/Users/me/Python/any2pandoc.py", line 59, in convert_word_file
    output = p2.communicate()[0]
AttributeError: 'int' object has no attribute 'communicate'

I get a markdown_file created but it's empty.

Thanks,

RD

ps - Daniel's works fine but I still don't learn to pipe :)

[toc] | [prev] | [next] | [standalone]

#65215

On Saturday, February 1, 2014 7:54:34 AM UTC-6, Rick Dooling wrote:
> On Saturday, February 1, 2014 6:54:09 AM UTC-6, Peter Otten wrote:
> 
> > Rick Dooling wrote:
> 
> > 
> 
> > 
> 
> > 
> 
> > > I spent half a day trying to convert this bash script (on Mac)
> 
> > 
> 
> > > 
> 
> > 
> 
> > > textutil -convert html $1 -stdout | pandoc -f html -t markdown -o $2
> 
> > 
> 
> > > 
> 
> > 
> 
> > > into Python using subprocess pipes.
> 
> > 
> 
> > > 
> 
> > 
> 
> > > It works if I save the above into a shell script called convert.sh and
> 
> > 
> 
> > > then do
> 
> > 
> 
> > > 
> 
> > 
> 
> > > subprocess.check_call(["convert.sh", file, markdown_file])
> 
> > 
> 
> > > 
> 
> > 
> 
> > > where file and markdown_file are variables.
> 
> > 
> 
> > > 
> 
> > 
> 
> > > But otherwise my piping attempts fail.
> 
> > 
> 
> > 
> 
> > 
> 
> > It is always a good idea to post your "best effort" failed attempt, if only 
> 
> > 
> 
> > to give us an idea of your level of expertise.
> 
> > 
> 
> > 
> 
> > 
> 
> > > Could someone show me how to pipe in subprocess. Yes, I've read the doc,
> 
> > 
> 
> > > especially
> 
> > 
> 
> > > 
> 
> > 
> 
> > > http://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline
> 
> > 
> 
> > > 
> 
> > 
> 
> > > But I'm a feeble hobbyist, not a computer scientist.
> 
> > 
> 
> > 
> 
> > 
> 
> > Try to convert the example from the above page
> 
> > 
> 
> > 
> 
> > 
> 
> > """
> 
> > 
> 
> > output=`dmesg | grep hda`
> 
> > 
> 
> > # becomes
> 
> > 
> 
> > p1 = Popen(["dmesg"], stdout=PIPE)
> 
> > 
> 
> > p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
> 
> > 
> 
> > p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
> 
> > 
> 
> > output = p2.communicate()[0]
> 
> > 
> 
> > """
> 
> > 
> 
> > 
> 
> > 
> 
> > to your usecase. Namely, replace
> 
> > 
> 
> > 
> 
> > 
> 
> > ["dmesg"] --> ["textutil", "-convert", "html", infile, "-stdout"]
> 
> > 
> 
> > ["grep", "hda"] --> ["pandoc", "-f", "html", "-t", "marktown", "-o",
> 
> > 
> 
> >                      outfile]
> 
> > 
> 
> > 
> 
> > 
> 
> > Don't forget to set
> 
> > 
> 
> > 
> 
> > 
> 
> > infile = ... 
> 
> > 
> 
> > outfile = ... 
> 
> > 
> 
> > 
> 
> > 
> 
> > to filenames (with absolute paths, to avoid one source of error).
> 
> > 
> 
> > If that doesn't work post the code you wrote along with the error messages.
> 
> 
> 
> p1 = subprocess.Popen(["textutil", "-convert", "html", file], stdout=subprocess.PIPE)
> 
> p2 = subprocess.check_call(["pandoc", "-f", "html", "-t", "markdown", "-o", markdown_file], stdin=p1.stdout, stdout=subprocess.PIPE)
> 
> p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
> 
> output = p2.communicate()[0]
> 
> 
> 
> Errors
> 
> 
> 
> Traceback (most recent call last):
> 
>   File "/Users/me/Python/any2pandoc.py", line 70, in <module>
> 
>     convert_word_file(file, markdown_file)
> 
>   File "/Users/me/Python/any2pandoc.py", line 59, in convert_word_file
> 
>     output = p2.communicate()[0]
> 
> AttributeError: 'int' object has no attribute 'communicate'
> 
> 
> 
> I get a markdown_file created but it's empty.
> 
> 
> 
> Thanks,
> 
> 
> 
> RD
> 
> 
> 
> ps - Daniel's works fine but I still don't learn to pipe :)

Okay, sorry. I fixed that obvious goof

    p1 = subprocess.Popen(["textutil", "-convert", "html", file], stdout=subprocess.PIPE)
    p2 = subprocess.Popen(["pandoc", "-f", "html", "-t", "markdown", "-o", markdown_file], stdin=p1.stdout, stdout=subprocess.PIPE)
    p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
    output = p2.communicate()[0]

Now I get no errors, but I still get a blank markdown file.

[toc] | [prev] | [next] | [standalone]

#65221

On Saturday, February 1, 2014 8:00:59 AM UTC-6, Rick Dooling wrote:
> On Saturday, February 1, 2014 7:54:34 AM UTC-6, Rick Dooling wrote:
> 
> > On Saturday, February 1, 2014 6:54:09 AM UTC-6, Peter Otten wrote:
> 
> > > Rick Dooling wrote:
> 
> > > > I spent half a day trying to convert this bash script (on Mac)
> > > > textutil -convert html $1 -stdout | pandoc -f html -t markdown -o $2
> > > > into Python using subprocess pipes.
> 
> > > > It works if I save the above into a shell script called convert.sh and
> > > > then do
> > 
> > > > subprocess.check_call(["convert.sh", file, markdown_file])
> 
> > > > where file and markdown_file are variables.
> 
> > > > But otherwise my piping attempts fail.
> 
> > > It is always a good idea to post your "best effort" failed attempt, if only to give us an idea of your level of expertise.
> > > > Could someone show me how to pipe in subprocess. Yes, I've read the doc,
> > > > especially http://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline
> 
> > > > But I'm a feeble hobbyist, not a computer scientist.
> 
> > > Try to convert the example from the above page
> 
> > > """
> 
> > > output=`dmesg | grep hda`
> 
> > > # becomes
> 
> > > p1 = Popen(["dmesg"], stdout=PIPE)
> 
> > > p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
> 
> > > p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
> 
> > > output = p2.communicate()[0]
> 
> > > """
> 
> > > to your usecase. Namely, replace
> 
> > > ["dmesg"] --> ["textutil", "-convert", "html", infile, "-stdout"]
> 
> > > ["grep", "hda"] --> ["pandoc", "-f", "html", "-t", "marktown", "-o"    outfile]
> 
> > > Don't forget to set
> 
> > > infile = ... 
> 
> > > outfile = ... 
> 
> > > to filenames (with absolute paths, to avoid one source of error).
> 
> > > If that doesn't work post the code you wrote along with the error messages.
> 
> > p1 = subprocess.Popen(["textutil", "-convert", "html", file], stdout=subprocess.PIPE)
> 
> > p2 = subprocess.check_call(["pandoc", "-f", "html", "-t", "markdown", "-o", markdown_file], stdin=p1.stdout, stdout=subprocess.PIPE)
> 
> > p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.

> > output = p2.communicate()[0]
> > Errors
> > Traceback (most recent call last):
> >   File "/Users/me/Python/any2pandoc.py", line 70, in <module>
> >     convert_word_file(file, markdown_file)
> >   File "/Users/me/Python/any2pandoc.py", line 59, in convert_word_file
> >     output = p2.communicate()[0]
> > AttributeError: 'int' object has no attribute 'communicate'
> > I get a markdown_file created but it's empty.
> > Thanks,
> > RD
> > ps - Daniel's works fine but I still don't learn to pipe :)
> Okay, sorry. I fixed that obvious goof

>     p1 = subprocess.Popen(["textutil", "-convert", "html", file], stdout=subprocess.PIPE)
> 
>     p2 = subprocess.Popen(["pandoc", "-f", "html", "-t", "markdown", "-o", markdown_file], stdin=p1.stdout, stdout=subprocess.PIPE)
> 
>     p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
> 
>     output = p2.communicate()[0]
> 
> Now I get no errors, but I still get a blank markdown file.

Okay, blank lines removed. Apologies. I didn't know Google inserted them.

RD

[toc] | [prev] | [next] | [standalone]

#65225

On 01/02/2014 15:35, Rick Dooling wrote:
>
> Okay, blank lines removed. Apologies. I didn't know Google inserted them.
>
> RD
>

No problem, the whole snag is people don't know about this flaw in this 
tool until they're told about it.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#65217

On 01/02/2014 13:54, Rick Dooling wrote:
> On Saturday, February 1, 2014 6:54:09 AM UTC-6, Peter Otten wrote:
>> Rick Dooling wrote:
>>
>>
>>
>>> I spent half a day trying to convert this bash script (on Mac)
>>
>>>
>>
>>> textutil -convert html $1 -stdout | pandoc -f html -t markdown -o $2
>>
>>>
>>
>>> into Python using subprocess pipes.
>>
>>>
>>
>>> It works if I save the above into a shell script called convert.sh and
>>
>>> then do
>>
>>>
>>
>>> subprocess.check_call(["convert.sh", file, markdown_file])
>>
>>>
>>
>>> where file and markdown_file are variables.
>>
>>>
>>
>>> But otherwise my piping attempts fail.
>>
>>
>>
>> It is always a good idea to post your "best effort" failed attempt, if only
>>
>> to give us an idea of your level of expertise.
>>
>>
>>
>>> Could someone show me how to pipe in subprocess. Yes, I've read the doc,
>>
>>> especially
>>
>>>
>>
>>> http://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline
>>
>>>
>>
>>> But I'm a feeble hobbyist, not a computer scientist.
>>
>>
>>
>> Try to convert the example from the above page
>>
>>
>>
>> """
>>
>> output=`dmesg | grep hda`
>>
>> # becomes
>>
>> p1 = Popen(["dmesg"], stdout=PIPE)
>>
>> p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
>>
>> p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
>>
>> output = p2.communicate()[0]
>>
>> """
>>
>>
>>
>> to your usecase. Namely, replace
>>
>>
>>
>> ["dmesg"] --> ["textutil", "-convert", "html", infile, "-stdout"]
>>
>> ["grep", "hda"] --> ["pandoc", "-f", "html", "-t", "marktown", "-o",
>>
>>                       outfile]
>>
>>
>>
>> Don't forget to set
>>
>>
>>
>> infile = ...
>>
>> outfile = ...
>>
>>
>>
>> to filenames (with absolute paths, to avoid one source of error).
>>
>> If that doesn't work post the code you wrote along with the error messages.

Would you please read and action this 
https://wiki.python.org/moin/GoogleGroupsPython to prevent us seeing the 
double line spacing above, thanks.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#65229

Rick Dooling wrote:

> On Saturday, February 1, 2014 6:54:09 AM UTC-6, Peter Otten wrote:

>> Try to convert the example from the above page
>> 
>> """
>> output=`dmesg | grep hda`
>> # becomes
>> p1 = Popen(["dmesg"], stdout=PIPE)
>> p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
>> p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
>> output = p2.communicate()[0]
>> """
>> 
>> to your usecase. Namely, replace
>> 
>> ["dmesg"] --> ["textutil", "-convert", "html", infile, "-stdout"]
>> ["grep", "hda"] --> ["pandoc", "-f", "html", "-t", "marktown", "-o",
>>                      outfile]
>> 
>> Don't forget to set
>> 
>> infile = ...
>> outfile = ...
>> 
>> to filenames (with absolute paths, to avoid one source of error).
>> If that doesn't work post the code you wrote along with the error
>> messages.
> 
> p1 = subprocess.Popen(["textutil", "-convert", "html", file],
> stdout=subprocess.PIPE) 
> p2 = subprocess.check_call(["pandoc", "-f",
> "html", "-t", "markdown", "-o", markdown_file], stdin=p1.stdout,
> stdout=subprocess.PIPE)
> p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
> output = p2.communicate()[0]
> 
> Errors
> 
> Traceback (most recent call last):
>   File "/Users/me/Python/any2pandoc.py", line 70, in <module>
>     convert_word_file(file, markdown_file)
>   File "/Users/me/Python/any2pandoc.py", line 59, in convert_word_file
>     output = p2.communicate()[0]
> AttributeError: 'int' object has no attribute 'communicate'
> 
> I get a markdown_file created but it's empty.

Well, you replaced the Popen() from the example with a check_call() which 
uses a Popen instance internally, but does not expose it. 

I recommend that you stick as closely to the example as possible until you 
have a working baseline version. I'd try

textutil = subprocess.Popen(
    ["textutil", "-convert", "html", file], 
    stdout=subprocess.PIPE)
pandoc = subprocess.Popen(
    ["pandoc", "-f", "html", "-t", "markdown", "-o", markdown_file],
    stdin=textutil.stdout)

textutil.stdout.close()
pandoc.communicate()

[toc] | [prev] | [next] | [standalone]

#65213

[Multipart message — attachments visible in raw view] — view raw

Try this:

from subprocess import check_output
import sys
check_output("textutil -convert html %s -stdout | pandoc -f html -t
markdown -o %s" % sys.argv[1:3], shell=True)




On Sat, Feb 1, 2014 at 7:19 AM, Rick Dooling <rpdooling@gmail.com> wrote:

> I spent half a day trying to convert this bash script (on Mac)
>
> textutil -convert html $1 -stdout | pandoc -f html -t markdown -o $2
>
> into Python using subprocess pipes.
>
> It works if I save the above into a shell script called convert.sh and
> then do
>
> subprocess.check_call(["convert.sh", file, markdown_file])
>
> where file and markdown_file are variables.
>
> But otherwise my piping attempts fail.
>
> Could someone show me how to pipe in subprocess. Yes, I've read the doc,
> especially
>
> http://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline
>
> But I'm a feeble hobbyist, not a computer scientist.
>
> Thanks
>
> RD
> --
> https://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web