Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #33251

Re: creating size-limited tar files

Show all headers | View raw

On Tue, Nov 13, 2012 at 9:07 AM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
> It'll look something like this:
>
>>>> p1 = subprocess.Popen(cmd1, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>>> p2 = subprocess.Popen(cmd2, shell=True, stdin=p1.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>>> p1.communicate()
> ('', '')
>>>> p2.communicate()
> ('', '')
>>>> p1.wait()
> 0
>>>> p2.wait()
> 0
>
> Note that there's a subtle potential for deadlock here.  During the
> p1.communicate() call, if the p2 output buffer fills up, then it will
> stop accepting input from p1 until p2.communicate() can be called, and
> then if that buffer also fills up, p1 will hang.  Additionally, if p2
> needs to wait on the parent process for some reason, then you end up
> effectively serializing the two processes.
>
> Solution would be to poll all the open-ended pipes in a select() loop
> instead of using communicate(), or perhaps make the two communicate
> calls simultaneously in separate threads.

Sorry, the example I gave above is wrong.  If you're calling
p1.communicate(), then you need to first remove the p1.stdout pipe
from the Popen object.  Otherwise, the communicate() call will try to
read data from it and may "steal" input from p2.  It should look more
like this:

>>> p1 = subprocess.Popen(cmd1, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> p2 = subprocess.Popen(cmd2, shell=True, stdin=p1.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> p1.stdout = None

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-07 17:13 +0000
  Re: creating size-limited tar files Neil Cerutti <neilc@norwich.edu> - 2012-11-07 18:40 +0000
  Re: creating size-limited tar files Alexander Blinne <news@blinne.net> - 2012-11-07 20:05 +0100
    Re: creating size-limited tar files Roy Smith <roy@panix.com> - 2012-11-07 15:32 -0500
      Re: creating size-limited tar files Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-11-07 21:52 +0000
      Re: creating size-limited tar files Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2012-11-07 23:15 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-08 10:11 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-08 10:29 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-09 10:39 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-13 10:31 +0000
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:07 -0700
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:25 -0700
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:30 -0700
      Re: creating size-limited tar files Kushal Kumaran <kushal.kumaran+python@gmail.com> - 2012-11-14 11:35 +0530
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-14 00:22 -0700
      Re: creating size-limited tar files Kushal Kumaran <kushal.kumaran+python@gmail.com> - 2012-11-14 14:21 +0530
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 11:52 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 15:56 +0000
      Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 11:10 -0500
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 16:16 +0000
  Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 11:33 -0500
  Re: creating size-limited tar files Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-11-14 20:43 +0000
  Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 15:57 -0500

csiph-web