Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #8522 > unrolled thread

Trying to chain processes together on a pipeline

Back to article view | Back to comp.lang.python

Contents

  Trying to chain processes together on a pipeline Andrew Berg <bahamutzero8825@gmail.com> - 2011-06-28 00:57 -0500
    Re: Trying to chain processes together on a pipeline Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2011-06-28 09:44 +0200
      Re: Trying to chain processes together on a pipeline Andrew Berg <bahamutzero8825@gmail.com> - 2011-06-28 03:09 -0500

#8522 — Trying to chain processes together on a pipeline

I'm working on an audio/video converter script (moving from bash to
Python for some extra functionality), and part of it is chaining the
audio decoder (FFmpeg) either into SoX to change the volume and then to
the Nero AAC encoder or directly into the Nero encoder. This is the
chunk of code from my working bash script to give an idea of what I'm
trying to accomplish (it's indented because it's nested inside a while
loop and an if statement):
>         if [ "$process_audio" = "true" ]
>         then
>             if [ $vol == 1.0 ]
>             then
>                 ffmpeg -i "${ifile_a}" -f wav - 2>$nul | neroaacenc
> -ignorelength -q 0.4 -if - -of ${prefix}${zero}${ep}.m4a
>             else
>                 # the pipeline-as-file option of sox fails on Windows
> 7, so I use the safe method since there's only one pipeline going into sox
>                 ffmpeg -i "${ifile_a}" -f sox - 2>$nul | sox -t sox -
> -t wav - vol $vol 2>$nul | neroaacenc -ignorelength -q 0.4 -if - -of
> ${prefix}${zero}${ep}.m4a
>             fi
>         else
>             echo "Audio skipped."
>         fi
This is pretty easy and straightforward in bash, but not so in Python.
This is what I have in Python (queue[position] points to an object I
create earlier that holds a bunch of info on what needs to be encoded -
input and output file names, command line options for the various
encoders used, and so forth), but clearly it has some problems:
>     try:
>         ffmpeg_proc = subprocess.Popen(queue[position].ffmpeg_cmd,
> stdout=subprocess.PIPE, stderr=os.devnull)
>     except WindowsError:
>         error_info = str(sys.exc_info()[1])
>         last_win_error_num = find_win_error_no(error_msg=error_info)
>         if last_win_error_num == '2': # Error 2 = 'The system cannot
> find the file specified'
>             logger.critical('Could not execute ' +
> queue[position].ffmpeg_exe + ': File not found.')
>         elif last_win_error_num == '193': # Error 193 = '%1 is not a
> valid Win32 application'
>             logger.critical('Could not execute ' +
> queue[position].ffmpeg_exe + ': It\'s not a valid Win32 application.')
>         break
>     if queue[position].vol != 1:
>         try:
>             sox_proc = subprocess.Popen(queue[position].sox_cmd,
> stdin=ffmpeg_proc.stdout, stdout=subprocess.PIPE, stderr=os.devnull)
>         except WindowsError:
>             error_info = str(sys.exc_info()[1])
>             last_win_error_num = find_win_error_no(error_msg=error_info)
>             if last_win_error_num == '2': # Error 2 = 'The system
> cannot find the file specified'
>                 logger.critical('Could not execute ' +
> queue[position].sox_exe + ': File not found.')
>             elif last_win_error_num == '193': # Error 193 = '%1 is not
> a valid Win32 application'
>                 logger.critical('Could not execute ' +
> queue[position].sox_exe + ': It\'s not a valid Win32 application.')
>             break
>         wav_pipe = sox_proc.stdout
>     else:
>         wav_pipe = ffmpeg_proc.stdout
>     try:
>         nero_aac_proc = subprocess.Popen(queue[position].nero_aac_cmd,
> stdin=wav_pipe)
>     except WindowsError:
>         error_info = str(sys.exc_info()[1])
>         last_win_error_num = find_win_error_no(error_msg=error_info)
>         if last_win_error_num == '2': # Error 2 = 'The system cannot
> find the file specified'
>             logger.critical('Could not execute ' +
> queue[position].sox_exe + ': File not found.')
>         elif last_win_error_num == '193': # Error 193 = '%1 is not a
> valid Win32 application'
>             logger.critical('Could not execute ' +
> queue[position].sox_exe + ': It\'s not a valid Win32 application.')
>         break
>     
>     ffmpeg_proc.wait()
>     if queue[position].vol != 1:
>         sox_proc.wait()
>     nero_aac_proc.wait()
>     break
Note: those break statements are there to break out of the while loop
this is in.
Firstly, that first assignment to ffmpeg_proc raises an exception:
> Traceback (most recent call last):
>   File "C:\Users\Bahamut\workspace\Disillusion\disillusion.py", line
> 288, in <module>
>     ffmpeg_proc = subprocess.Popen(queue[position].ffmpeg_cmd,
> stdout=subprocess.PIPE, stderr=os.devnull)
>   File "C:\Python32\lib\subprocess.py", line 700, in __init__
>     errread, errwrite) = self._get_handles(stdin, stdout, stderr)
>   File "C:\Python32\lib\subprocess.py", line 861, in _get_handles
>     errwrite = msvcrt.get_osfhandle(stderr.fileno())
> AttributeError: 'str' object has no attribute 'fileno'
I'm not really sure what it's complaining about since the exception
propagates from the msvcrt module through the subprocess module into my
program. I'm thinking it has to do my stderr assignment, but if that's
not right, I don't know what is.
Secondly, there are no Popen.stdout.close() calls because I'm not sure
where to put them.
Thirdly, I have nearly identical except WindowsError: blocks repeated -
I'm sure I can avoid this with decorators as suggested in a recent
thread, but I haven't learned decorators yet. This code isn't very
complete; I'm trying to get it to work in ideal conditions first, then
worry about how to handle failure.

Using Python 3 on Windows 7.

[toc] | [next] | [standalone]

#8526

Am 28.06.2011 07:57 schrieb Andrew Berg:
> I'm working on an audio/video converter script (moving from bash to
> Python for some extra functionality), and part of it is chaining the
> audio decoder (FFmpeg) either into SoX to change the volume and then to
> the Nero AAC encoder or directly into the Nero encoder. This is the
> chunk of code from my working bash script to give an idea of what I'm
> trying to accomplish (it's indented because it's nested inside a while
> loop and an if statement):
>>          if [ "$process_audio" = "true" ]
>>          then
>>              if [ $vol == 1.0 ]
>>              then
>>                  ffmpeg -i "${ifile_a}" -f wav - 2>$nul | neroaacenc
>> -ignorelength -q 0.4 -if - -of ${prefix}${zero}${ep}.m4a
>>              else
>>                  # the pipeline-as-file option of sox fails on Windows
>> 7, so I use the safe method since there's only one pipeline going into sox
>>                  ffmpeg -i "${ifile_a}" -f sox - 2>$nul | sox -t sox -
>> -t wav - vol $vol 2>$nul | neroaacenc -ignorelength -q 0.4 -if - -of
>> ${prefix}${zero}${ep}.m4a
>>              fi
>>          else
>>              echo "Audio skipped."
>>          fi
> This is pretty easy and straightforward in bash, but not so in Python.
> This is what I have in Python (queue[position] points to an object I
> create earlier that holds a bunch of info on what needs to be encoded -
> input and output file names, command line options for the various
> encoders used, and so forth), but clearly it has some problems:
>>      try:
>>          ffmpeg_proc = subprocess.Popen(queue[position].ffmpeg_cmd,
>> stdout=subprocess.PIPE, stderr=os.devnull)
>>      except WindowsError:
>>          error_info = str(sys.exc_info()[1])
>>          last_win_error_num = find_win_error_no(error_msg=error_info)
>>          if last_win_error_num == '2': # Error 2 = 'The system cannot
>> find the file specified'
>>              logger.critical('Could not execute ' +
>> queue[position].ffmpeg_exe + ': File not found.')
>>          elif last_win_error_num == '193': # Error 193 = '%1 is not a
>> valid Win32 application'
>>              logger.critical('Could not execute ' +
>> queue[position].ffmpeg_exe + ': It\'s not a valid Win32 application.')
>>          break
>>      if queue[position].vol != 1:
>>          try:
>>              sox_proc = subprocess.Popen(queue[position].sox_cmd,
>> stdin=ffmpeg_proc.stdout, stdout=subprocess.PIPE, stderr=os.devnull)
>>          except WindowsError:
>>              error_info = str(sys.exc_info()[1])
>>              last_win_error_num = find_win_error_no(error_msg=error_info)
>>              if last_win_error_num == '2': # Error 2 = 'The system
>> cannot find the file specified'
>>                  logger.critical('Could not execute ' +
>> queue[position].sox_exe + ': File not found.')
>>              elif last_win_error_num == '193': # Error 193 = '%1 is not
>> a valid Win32 application'
>>                  logger.critical('Could not execute ' +
>> queue[position].sox_exe + ': It\'s not a valid Win32 application.')
>>              break
>>          wav_pipe = sox_proc.stdout
>>      else:
>>          wav_pipe = ffmpeg_proc.stdout
>>      try:
>>          nero_aac_proc = subprocess.Popen(queue[position].nero_aac_cmd,
>> stdin=wav_pipe)
>>      except WindowsError:
>>          error_info = str(sys.exc_info()[1])
>>          last_win_error_num = find_win_error_no(error_msg=error_info)
>>          if last_win_error_num == '2': # Error 2 = 'The system cannot
>> find the file specified'
>>              logger.critical('Could not execute ' +
>> queue[position].sox_exe + ': File not found.')
>>          elif last_win_error_num == '193': # Error 193 = '%1 is not a
>> valid Win32 application'
>>              logger.critical('Could not execute ' +
>> queue[position].sox_exe + ': It\'s not a valid Win32 application.')
>>          break
>>
>>      ffmpeg_proc.wait()
>>      if queue[position].vol != 1:
>>          sox_proc.wait()
>>      nero_aac_proc.wait()
>>      break
> Note: those break statements are there to break out of the while loop
> this is in.
> Firstly, that first assignment to ffmpeg_proc raises an exception:
>> Traceback (most recent call last):
>>    File "C:\Users\Bahamut\workspace\Disillusion\disillusion.py", line
>> 288, in<module>
>>      ffmpeg_proc = subprocess.Popen(queue[position].ffmpeg_cmd,
>> stdout=subprocess.PIPE, stderr=os.devnull)
>>    File "C:\Python32\lib\subprocess.py", line 700, in __init__
>>      errread, errwrite) = self._get_handles(stdin, stdout, stderr)
>>    File "C:\Python32\lib\subprocess.py", line 861, in _get_handles
>>      errwrite = msvcrt.get_osfhandle(stderr.fileno())
>> AttributeError: 'str' object has no attribute 'fileno'
> I'm not really sure what it's complaining about since the exception
> propagates from the msvcrt module through the subprocess module into my
> program. I'm thinking it has to do my stderr assignment, but if that's
> not right, I don't know what is.
> Secondly, there are no Popen.stdout.close() calls because I'm not sure
> where to put them.

> Thirdly, I have nearly identical except WindowsError: blocks repeated -
> I'm sure I can avoid this with decorators as suggested in a recent
> thread, but I haven't learned decorators yet.

There is a variety of possibilities. As you are a starter, a function is 
prefecty fine, as Peter already pointed out. Then you have something like

def handle_winerr(exc):
     error_info = str(exc)
     last_win_error_num = find_win_error_no(error_msg=error_info)
     if last_win_error_num == '2': # Error 2 = 'The system cannot find 
the file specified'
         logger.critical('Could not execute ' +
             queue[position].sox_exe + ': File not found.')
     elif last_win_error_num == '193': # Error 193 = '%1 is not a valid 
Win32 application'
         logger.critical('Could not execute ' +
             queue[position].sox_exe + ': It\'s not a valid Win32 
application.')

try:
     <do_stuff>
except WindowsError:
     exc = sys.exc_info()[1]
     handle_winerr(exc)
     break


On the next step, you could write like this:

try:
     <do_stuff>
except WindowsError, exc:
     handle_winerr(exc)
     break


The way you work with the exception is not the very best - instead of 
parsing the stringified exception, you better would trigger on 
exc.winerror (it is an integer with the error number).

Or, even better, just pas the error information contained in the exception:

def handle_winerr(exc):
     logger.critical('Could not execute %s: %s' %
         (queue[position].sox_exe, exc.strerror))


Next step: If you always break out of the loop when there is an error, 
you could as well do the error handling outside the loop:


try:
     while <whatever>:
         <do_stuff>
except WindowsError, exc:
     handle_winerr(exc)



In the following, I'm pointing out some more or less advanced techniques 
which you also could use.

If you want to, you can create a context manager dealing with the stuff:

import contextlib

@contextlib.contextmanager
def winerr_handling():
     try:
         yield None
     except WindowsError, exc:
         handle_winerr(exc) # <-- the function from above

and then write

with winerr_handling():
     while <whatever>:
         <do_stuff>

And, if you absolutely want a decorator, you can do so:

def make_exc_ctx(hdlr):
     from functools import wraps
     @wraps(hdlr)
     def wrapper():
         try:
             yield None
         except WindowsError, exc:
             hdlr(exc) # <-- the function from above


and then

@make_exc_ctx
def winerr_handling(exc):
     logger.critical('Could not execute %s: %s' %
         (queue[position].sox_exe, exc.strerror))

with winerr_handling():
     while <whatever>:
         <do_stuff>


HTH,

Thomas

[toc] | [prev] | [next] | [standalone]

#8528

On 2011.06.28 02:44 AM, Thomas Rachel wrote:
> The way you work with the exception is not the very best - instead of 
> parsing the stringified exception, you better would trigger on 
> exc.winerror (it is an integer with the error number).
>
> Or, even better, just pas the error information contained in the exception:
>
> def handle_winerr(exc):
>      logger.critical('Could not execute %s: %s' %
>          (queue[position].sox_exe, exc.strerror))
I didn't see winerror and strerror in the docs before, but another look
and they are indeed documented. I brought up Windows error codes before,
and I'm surprised no one pointed this out. Thanks for that.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web