Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #8617 > unrolled thread

Re: Trying to chain processes together on a pipeline

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Trying to chain processes together on a pipeline Andrew Berg <bahamutzero8825@gmail.com> - 2011-06-30 20:40 -0500
    Re: Trying to chain processes together on a pipeline Peter Otten <__peter__@web.de> - 2011-07-01 09:26 +0200
      Re: Trying to chain processes together on a pipeline Andrew Berg <bahamutzero8825@gmail.com> - 2011-07-01 03:02 -0500
      Re: Trying to chain processes together on a pipeline Chris Rebert <clp2@rebertia.com> - 2011-07-01 01:24 -0700

#8617 — Re: Trying to chain processes together on a pipeline

Okay, so I've refactored those except WindowsError blocks into calls to
a function and fixed the os.devnull bug, but I still can't get the
triple chain working. I added calls to ffmpeg_proc.stdout.close() and
sox_proc.stdout.close(), but I really am not sure where to put them. The
following code works if SoX isn't part of the chain (that is, if vol ==
1), but not otherwise (the Nero encoder says "truncation error" after it
finishes; the same error I get if omit the close() calls):
> while True:
>     if queue[position].audio:
>         if queue[position].vol != 1:
>             queue[position].ffmpeg_cmd = [
>             queue[position].ffmpeg_exe,
>             '-i', queue[position].ifile,
>             '-f', 'sox', '-'
>             ]
>             queue[position].sox_cmd = [
>             queue[position].sox_exe,
>             '-t', 'sox', '-',
>             '-t', 'wav', '-',
>             'vol', str(queue[position].vol)
>             ]
>         else:
>             queue[position].ffmpeg_cmd = [
>             queue[position].ffmpeg_exe,
>             '-i', queue[position].ifile,
>             '-f', 'wav', '-'
>             ]
>         queue[position].nero_aac_cmd = [
>         queue[position].nero_aac_exe,
>         '-ignorelength',
>         '-q', str(queue[position].aac_quality),
>         '-if', '-',
>         '-of', queue[position].outdir + queue[position].prefix + '.m4a'
>         ]
>     print(queue[position].ffmpeg_cmd)
>     print(queue[position].sox_cmd)
>     print(queue[position].nero_aac_cmd)
>     try:
>         ffmpeg_proc = subprocess.Popen(queue[position].ffmpeg_cmd,
> stdout=subprocess.PIPE, stderr=open(os.devnull, 'w'))
>     except WindowsError as exc:
>         log_windows_error(exc, queue[position].ffmpeg_cmd, 'critical')
>         break
>     if queue[position].vol != 1:
>         try:
>             sox_proc = subprocess.Popen(queue[position].sox_cmd,
> stdin=ffmpeg_proc.stdout, stdout=subprocess.PIPE,
> stderr=open(os.devnull, 'w'))
>             wav_pipe = sox_proc.stdout
>         except WindowsError as exc:
>             log_windows_error(exc, queue[position].sox_cmd, 'critical')
>             break
>     else:
>         wav_pipe = ffmpeg_proc.stdout
>     try:
>         nero_aac_proc = subprocess.Popen(queue[position].nero_aac_cmd,
> stdin=wav_pipe)
>     except WindowsError as exc:
>         log_windows_error(exc, queue[position].nero_aac_cmd, 'critical')
>         break
>     finally:
>         ffmpeg_proc.stdout.close()
>         if queue[position].vol != 1:
>             sox_proc.stdout.close()
>     ffmpeg_proc.wait()
>     if queue[position].vol != 1:
>         sox_proc.wait()
>     nero_aac_proc.wait()
>     break

[toc] | [next] | [standalone]

#8627

Andrew Berg wrote:

> Okay, so I've refactored those except WindowsError blocks into calls to
> a function and fixed the os.devnull bug, but I still can't get the
> triple chain working. I added calls to ffmpeg_proc.stdout.close() and
> sox_proc.stdout.close(), but I really am not sure where to put them. The
> following code works if SoX isn't part of the chain (that is, if vol ==
> 1), but not otherwise (the Nero encoder says "truncation error" after it
> finishes; the same error I get if omit the close() calls):

I can't reproduce your setup, but I'd try using communicate() instead of 
wait() and close().

[toc] | [prev] | [next] | [standalone]

#8630

On 2011.07.01 02:26 AM, Peter Otten wrote:
> I can't reproduce your setup, but I'd try using communicate() instead of 
> wait() and close().
I don't really know what communicate() does. The docs don't give much
info or any examples (that explain communicate() anyway), and don't say
when communicate() is useful.

[toc] | [prev] | [next] | [standalone]

#8631

On Fri, Jul 1, 2011 at 1:02 AM, Andrew Berg <bahamutzero8825@gmail.com> wrote:
> On 2011.07.01 02:26 AM, Peter Otten wrote:
>> I can't reproduce your setup, but I'd try using communicate() instead of
>> wait() and close().
> I don't really know what communicate() does.

"Read data from stdout and stderr, until end-of-file is reached. Wait
for process to terminate."
It then returns the read data as two strings. It's pretty straightforward.

> The docs don't give much
> info or any examples (that explain communicate() anyway), and don't say
> when communicate() is useful.

They are slightly roundabout in that respect. The warnings for .wait()
and .stdout/err explain communicate()'s utility:
"Warning: This will deadlock when using stdout=PIPE and/or stderr=PIPE
and the child process generates enough output to a pipe such that it
blocks waiting for the OS pipe buffer to accept more data. ***Use
communicate() to avoid that.***" [emphasis added]

Cheers,
Chris
--
http://rebertia.com

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web