Path: csiph.com!usenet.pasdenom.info!dedibox.gegeweb.org!gegeweb.eu!nntpfeed.proxad.net!proxad.net!feeder1-2.proxad.net!usenet-fr.net!nerim.net!novso.com!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Date: Thu, 13 Sep 2012 17:34:29 +0200 (CEST)
From: Jean-Michel Pichavant <jeanmichel@sequans.com>
To: paulstaten@gmail.com
In-Reply-To: <d49e1e71-afe2-4aab-8c67-3e2eea6533a6@googlegroups.com>
Subject: Re: subprocess call is not waiting.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Cc: python-list@python.org
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.612.1347550453.27098.python-list@python.org>
Lines: 44
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:29044

----- Original Message -----
> I have a subprocess.call which tries to download a data from a remote
> server using HTAR. I put the call in a while loop, which tests to
> see if the download was successful, and if not, loops back around up
> to five times, just in case my internet connection has a hiccup.
> 
> Subprocess.call is supposed to wait.
> 
> But it doesn't work as intended. The loop quickly runs 5 times,
> starting a new htar command each time. After five times around, my
> program tells me my download failed, because the target file doesn't
> yet exist. But it turns out that the download is still
> happening---five times.
> 
> When I run htar from the shell, I don't get a shell prompt again
> until after the download is complete. How come control is returned
> to python before the htar command is through?
> 
> I've tried using Popen with wait and/or communicate, but no waiting
> ever happens. This is troublesome not only because I don't get to
> post process my data, but because when I run this script for
> multiple datasets (checking to see whether I have local copies), I
> quickly get a "Too many open files" error. (I began working on that
> by trying to use Popopen with fds_close, etc.)
> 
> Should I just go back to os.system?
> --
> http://mail.python.org/mailman/listinfo/python-list
> 

A related subset of code would be useful. 

You can use subprocess.PIPE to redirect stdout & stderr et get them with communicate, something like:

proc = subprocess.Popen(['whatever'], stdout=subprocess.PIPE, stdout=subprocess.PIPE)
stdout, stderr = proc.communicate()
print stdout
print stderr

Just by looking at stdout and stderr, you should be able to see why htar is returning so fast.

JM

PS : if you see nothing wrong, is it possible that htar runs asynchronously ?