Path: csiph.com!usenet.pasdenom.info!dedibox.gegeweb.org!gegeweb.eu!nntpfeed.proxad.net!proxad.net!feeder1-2.proxad.net!usenet-fr.net!nerim.net!novso.com!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.003 X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'output': 0.04; "'')": 0.07; 'reason,': 0.07; 'subject:files': 0.09; 'subtle': 0.09; 'threads.': 0.09; "('',": 0.16; 'called,': 0.16; 'deadlock': 0.16; 'pipes': 0.16; 'poll': 0.16; 'processes.': 0.16; 'subprocess': 0.16; 'ugly.': 0.16; 'wrote:': 0.17; 'accepting': 0.18; '>>>': 0.18; 'input': 0.18; 'bit': 0.21; 'this:': 0.23; 'header:In-Reply- To:1': 0.25; 'am,': 0.27; 'separate': 0.27; 'wonder': 0.27; 'message-id:@mail.gmail.com': 0.27; 'parent': 0.29; 'subject:size': 0.29; 'to:addr:python-list': 0.33; 'received:google.com': 0.34; 'nov': 0.35; 'received:209.85': 0.35; 'something': 0.35; 'but': 0.36; 'two': 0.37; 'communicate': 0.37; 'received:209': 0.37; 'subject:: ': 0.38; 'some': 0.38; 'instead': 0.39; 'to:addr:python.org': 0.39; 'subject:-': 0.40; 'header:Received:5': 0.40; 'end': 0.40; 'andrea': 0.84; 'thing,': 0.84; 'to:name:python': 0.84; 'subject:limited': 0.93 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=fI6JQOrHaxMI2tBASJ6hgdfB+hXDKbq4e2xcNRKiteg=; b=Q7atQx0yYm01AIcOdXMkT7GIP2+1zR6FhGGPa8ZbKvk+iuCmdX2UTxnlGzEYykdZcf tcFeCmgr6xnEY5sTnSMaXF+7OnSyk0A5e0ALc8tn36udgYWtUUaHHyqJTYMqxliMEZa6 vJrM3WUBRg25lr5hZk0ySI7ShmLObVAYQan87m577OPfsqrj15MQ5+BEbEvc1TUkKiR6 uuHGG/oIpI0jlrx+xejIypynEo+cFL8p64aKMrWL7aLc8RLVmFxvsIkPCtti6Fm360es yWe34CcD3QNC9r8sznFiNPWlryAgsbLNdaNbv8Te//pGfovTSrEDfscoagmJy/Oe127b Ls+A== MIME-Version: 1.0 In-Reply-To: References: <509ab0fa$0$6636$9b4e6d93@newsspool2.arcor-online.net> <509AD812.2060605@gmail.com> From: Ian Kelly Date: Tue, 13 Nov 2012 09:07:15 -0700 Subject: Re: creating size-limited tar files To: Python Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 28 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1352822868 news.xs4all.nl 6893 [2001:888:2000:d::a6]:50153 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:33247 On Tue, Nov 13, 2012 at 3:31 AM, andrea crotti wrote: > but it's a bit ugly. I wonder if I can use the subprocess PIPEs to do > the same thing, is it going to be as fast and work in the same way?? It'll look something like this: >>> p1 = subprocess.Popen(cmd1, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) >>> p2 = subprocess.Popen(cmd2, shell=True, stdin=p1.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE) >>> p1.communicate() ('', '') >>> p2.communicate() ('', '') >>> p1.wait() 0 >>> p2.wait() 0 Note that there's a subtle potential for deadlock here. During the p1.communicate() call, if the p2 output buffer fills up, then it will stop accepting input from p1 until p2.communicate() can be called, and then if that buffer also fills up, p1 will hang. Additionally, if p2 needs to wait on the parent process for some reason, then you end up effectively serializing the two processes. Solution would be to poll all the open-ended pipes in a select() loop instead of using communicate(), or perhaps make the two communicate calls simultaneously in separate threads.