Re: creating size-limited tar files

Path	csiph.com!usenet.pasdenom.info!news.albasani.net!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path	<ian.g.kelly@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.004
X-Spam-Evidence	'H': 0.99; 'S': 0.00; 'output': 0.04; "'')": 0.07; 'reason,': 0.07; 'subject:files': 0.09; 'subtle': 0.09; 'threads.': 0.09; "('',": 0.16; 'called,': 0.16; 'deadlock': 0.16; 'pipes': 0.16; 'poll': 0.16; 'processes.': 0.16; 'wrote:': 0.17; 'accepting': 0.18; '>>>': 0.18; 'input': 0.18; 'otherwise,': 0.20; 'object.': 0.22; 'pipe': 0.22; 'sorry,': 0.22; 'example': 0.23; 'this:': 0.23; 'header:In-Reply-To:1': 0.25; 'am,': 0.27; 'separate': 0.27; 'message-id:@mail.gmail.com': 0.27; '>>>>': 0.29; 'parent': 0.29; 'subject:size': 0.29; 'received:209.85.215.46': 0.30; 'to:addr:python-list': 0.33; 'received:google.com': 0.34; 'nov': 0.35; 'received:209.85': 0.35; 'something': 0.35; 'should': 0.36; 'two': 0.37; 'communicate': 0.37; 'received:209': 0.37; 'data': 0.37; 'subject:: ': 0.38; 'some': 0.38; 'instead': 0.39; 'to:addr:python.org': 0.39; 'subject:-': 0.40; 'header:Received:5': 0.40; 'end': 0.40; 'remove': 0.61; 'first': 0.61; 'more': 0.63; 'gave': 0.65; 'to:name:python': 0.84; 'subject:limited': 0.93
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=v/BSbTiKmEsTYSh4Vd5FjQN3jauQT1HZPR9RR54eltc=; b=Iqidw0NlgkVK1jUXxLZauZ/GwGofPMl4VfswMajUV6lUfgYZSPIFq0vwZX6rDJ77Pr LKmrGsyHgs4OVtnTvYfX1jBBvCI8nubQn5JAhZzWf2yYhX/4MazEGVoN8nSj/8aznFZ6 OJ1Spi0JcjG77xreaWHmVuYDqh7/zsA5ys52arwFLo+U4BwmTRbikRJW+eubRKDKbs8r JcoLfREe31xR1KCczpcA4sBTrBTdzpOx8LkUq5CGrc0X6gGWm/Hj7YdIYED1ojWRbty5 sbUokOnbU+dqyk5fIg8i1v6u3See4ZLSbe0qmnUY6FAV9owOY3kl/KZEA+pvO1Vd2v94 C5jQ==
MIME-Version	1.0
In-Reply-To	<CALwzidnrTcSK+Z0cGf5WpTN2sLqVgbpbt2i1DcdUFnGQYEvKQg@mail.gmail.com>
References	<mailman.3381.1352308430.27098.python-list@python.org> <509ab0fa$0$6636$9b4e6d93@newsspool2.arcor-online.net> <roy-2C9B4D.15322307112012@news.panix.com> <509AD812.2060605@gmail.com> <CAHVvXxTH7WrfT41GDj15f9FS4eguBF2dj4Khpu_vw2TWMkvPYg@mail.gmail.com> <CAF_E5JbY71J3ix9PiXu7wvvkzc_aG9ZtdwKkuxXXAU-SFeRDsw@mail.gmail.com> <CAF_E5JYJ7Fw4b4z81454k9HwmjZ49qjx48fC_f+skcphCDNUeA@mail.gmail.com> <CAF_E5Jas3ObdX=Mhusj63R_Xcm0e1JwjPEuOgZoggON6-cttjA@mail.gmail.com> <CAF_E5JZFJ+TPmF0LUEFU8AVqbOWfJbH9Yy60CKkcrDNnWATiVg@mail.gmail.com> <CALwzidnrTcSK+Z0cGf5WpTN2sLqVgbpbt2i1DcdUFnGQYEvKQg@mail.gmail.com>
From	Ian Kelly <ian.g.kelly@gmail.com>
Date	Tue, 13 Nov 2012 09:25:20 -0700
Subject	Re: creating size-limited tar files
To	Python <python-list@python.org>
Content-Type	text/plain; charset=ISO-8859-1
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.15
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.3634.1352823951.27098.python-list@python.org> (permalink)
Lines	34
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1352823951 news.xs4all.nl 6845 [2001:888:2000:d::a6]:36859
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:33251

Show key headers only | View raw

On Tue, Nov 13, 2012 at 9:07 AM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
> It'll look something like this:
>
>>>> p1 = subprocess.Popen(cmd1, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>>> p2 = subprocess.Popen(cmd2, shell=True, stdin=p1.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>>> p1.communicate()
> ('', '')
>>>> p2.communicate()
> ('', '')
>>>> p1.wait()
> 0
>>>> p2.wait()
> 0
>
> Note that there's a subtle potential for deadlock here.  During the
> p1.communicate() call, if the p2 output buffer fills up, then it will
> stop accepting input from p1 until p2.communicate() can be called, and
> then if that buffer also fills up, p1 will hang.  Additionally, if p2
> needs to wait on the parent process for some reason, then you end up
> effectively serializing the two processes.
>
> Solution would be to poll all the open-ended pipes in a select() loop
> instead of using communicate(), or perhaps make the two communicate
> calls simultaneously in separate threads.

Sorry, the example I gave above is wrong.  If you're calling
p1.communicate(), then you need to first remove the p1.stdout pipe
from the Popen object.  Otherwise, the communicate() call will try to
read data from it and may "steal" input from p2.  It should look more
like this:

>>> p1 = subprocess.Popen(cmd1, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> p2 = subprocess.Popen(cmd2, shell=True, stdin=p1.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> p1.stdout = None

Thread

creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-07 17:13 +0000
  Re: creating size-limited tar files Neil Cerutti <neilc@norwich.edu> - 2012-11-07 18:40 +0000
  Re: creating size-limited tar files Alexander Blinne <news@blinne.net> - 2012-11-07 20:05 +0100
    Re: creating size-limited tar files Roy Smith <roy@panix.com> - 2012-11-07 15:32 -0500
      Re: creating size-limited tar files Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-11-07 21:52 +0000
      Re: creating size-limited tar files Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2012-11-07 23:15 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-08 10:11 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-08 10:29 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-09 10:39 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-13 10:31 +0000
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:07 -0700
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:25 -0700
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:30 -0700
      Re: creating size-limited tar files Kushal Kumaran <kushal.kumaran+python@gmail.com> - 2012-11-14 11:35 +0530
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-14 00:22 -0700
      Re: creating size-limited tar files Kushal Kumaran <kushal.kumaran+python@gmail.com> - 2012-11-14 14:21 +0530
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 11:52 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 15:56 +0000
      Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 11:10 -0500
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 16:16 +0000
  Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 11:33 -0500
  Re: creating size-limited tar files Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-11-14 20:43 +0000
  Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 15:57 -0500

csiph-web