Re: creating size-limited tar files

Path	csiph.com!usenet.pasdenom.info!dedibox.gegeweb.org!gegeweb.eu!nntpfeed.proxad.net!proxad.net!feeder1-2.proxad.net!usenet-fr.net!nerim.net!novso.com!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path	<ian.g.kelly@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.003
X-Spam-Evidence	'H': 0.99; 'S': 0.00; 'output': 0.04; "'')": 0.07; 'reason,': 0.07; 'subject:files': 0.09; 'subtle': 0.09; 'threads.': 0.09; "('',": 0.16; 'called,': 0.16; 'deadlock': 0.16; 'pipes': 0.16; 'poll': 0.16; 'processes.': 0.16; 'subprocess': 0.16; 'ugly.': 0.16; 'wrote:': 0.17; 'accepting': 0.18; '>>>': 0.18; 'input': 0.18; 'bit': 0.21; 'this:': 0.23; 'header:In-Reply- To:1': 0.25; 'am,': 0.27; 'separate': 0.27; 'wonder': 0.27; 'message-id:@mail.gmail.com': 0.27; 'parent': 0.29; 'subject:size': 0.29; 'to:addr:python-list': 0.33; 'received:google.com': 0.34; 'nov': 0.35; 'received:209.85': 0.35; 'something': 0.35; 'but': 0.36; 'two': 0.37; 'communicate': 0.37; 'received:209': 0.37; 'subject:: ': 0.38; 'some': 0.38; 'instead': 0.39; 'to:addr:python.org': 0.39; 'subject:-': 0.40; 'header:Received:5': 0.40; 'end': 0.40; 'andrea': 0.84; 'thing,': 0.84; 'to:name:python': 0.84; 'subject:limited': 0.93
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=fI6JQOrHaxMI2tBASJ6hgdfB+hXDKbq4e2xcNRKiteg=; b=Q7atQx0yYm01AIcOdXMkT7GIP2+1zR6FhGGPa8ZbKvk+iuCmdX2UTxnlGzEYykdZcf tcFeCmgr6xnEY5sTnSMaXF+7OnSyk0A5e0ALc8tn36udgYWtUUaHHyqJTYMqxliMEZa6 vJrM3WUBRg25lr5hZk0ySI7ShmLObVAYQan87m577OPfsqrj15MQ5+BEbEvc1TUkKiR6 uuHGG/oIpI0jlrx+xejIypynEo+cFL8p64aKMrWL7aLc8RLVmFxvsIkPCtti6Fm360es yWe34CcD3QNC9r8sznFiNPWlryAgsbLNdaNbv8Te//pGfovTSrEDfscoagmJy/Oe127b Ls+A==
MIME-Version	1.0
In-Reply-To	<CAF_E5JZFJ+TPmF0LUEFU8AVqbOWfJbH9Yy60CKkcrDNnWATiVg@mail.gmail.com>
References	<mailman.3381.1352308430.27098.python-list@python.org> <509ab0fa$0$6636$9b4e6d93@newsspool2.arcor-online.net> <roy-2C9B4D.15322307112012@news.panix.com> <509AD812.2060605@gmail.com> <CAHVvXxTH7WrfT41GDj15f9FS4eguBF2dj4Khpu_vw2TWMkvPYg@mail.gmail.com> <CAF_E5JbY71J3ix9PiXu7wvvkzc_aG9ZtdwKkuxXXAU-SFeRDsw@mail.gmail.com> <CAF_E5JYJ7Fw4b4z81454k9HwmjZ49qjx48fC_f+skcphCDNUeA@mail.gmail.com> <CAF_E5Jas3ObdX=Mhusj63R_Xcm0e1JwjPEuOgZoggON6-cttjA@mail.gmail.com> <CAF_E5JZFJ+TPmF0LUEFU8AVqbOWfJbH9Yy60CKkcrDNnWATiVg@mail.gmail.com>
From	Ian Kelly <ian.g.kelly@gmail.com>
Date	Tue, 13 Nov 2012 09:07:15 -0700
Subject	Re: creating size-limited tar files
To	Python <python-list@python.org>
Content-Type	text/plain; charset=ISO-8859-1
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.15
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.3630.1352822868.27098.python-list@python.org> (permalink)
Lines	28
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1352822868 news.xs4all.nl 6893 [2001:888:2000:d::a6]:50153
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:33247

Show key headers only | View raw

On Tue, Nov 13, 2012 at 3:31 AM, andrea crotti
<andrea.crotti.0@gmail.com> wrote:
> but it's a bit ugly.  I wonder if I can use the subprocess PIPEs to do
> the same thing, is it going to be as fast and work in the same way??

It'll look something like this:

>>> p1 = subprocess.Popen(cmd1, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> p2 = subprocess.Popen(cmd2, shell=True, stdin=p1.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> p1.communicate()
('', '')
>>> p2.communicate()
('', '')
>>> p1.wait()
0
>>> p2.wait()
0

Note that there's a subtle potential for deadlock here.  During the
p1.communicate() call, if the p2 output buffer fills up, then it will
stop accepting input from p1 until p2.communicate() can be called, and
then if that buffer also fills up, p1 will hang.  Additionally, if p2
needs to wait on the parent process for some reason, then you end up
effectively serializing the two processes.

Solution would be to poll all the open-ended pipes in a select() loop
instead of using communicate(), or perhaps make the two communicate
calls simultaneously in separate threads.

Thread

creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-07 17:13 +0000
  Re: creating size-limited tar files Neil Cerutti <neilc@norwich.edu> - 2012-11-07 18:40 +0000
  Re: creating size-limited tar files Alexander Blinne <news@blinne.net> - 2012-11-07 20:05 +0100
    Re: creating size-limited tar files Roy Smith <roy@panix.com> - 2012-11-07 15:32 -0500
      Re: creating size-limited tar files Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-11-07 21:52 +0000
      Re: creating size-limited tar files Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2012-11-07 23:15 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-08 10:11 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-08 10:29 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-09 10:39 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-13 10:31 +0000
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:07 -0700
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:25 -0700
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:30 -0700
      Re: creating size-limited tar files Kushal Kumaran <kushal.kumaran+python@gmail.com> - 2012-11-14 11:35 +0530
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-14 00:22 -0700
      Re: creating size-limited tar files Kushal Kumaran <kushal.kumaran+python@gmail.com> - 2012-11-14 14:21 +0530
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 11:52 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 15:56 +0000
      Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 11:10 -0500
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 16:16 +0000
  Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 11:33 -0500
  Re: creating size-limited tar files Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-11-14 20:43 +0000
  Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 15:57 -0500

csiph-web