Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #29048 > unrolled thread

Re: subprocess call is not waiting.

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: subprocess call is not waiting. MRAB <python@mrabarnett.plus.com> - 2012-09-13 16:46 +0100
    Re: subprocess call is not waiting. paulstaten@gmail.com - 2012-09-13 11:36 -0700
      Re: subprocess call is not waiting. Chris Rebert <clp2@rebertia.com> - 2012-09-13 22:24 -0700
    Re: subprocess call is not waiting. paulstaten@gmail.com - 2012-09-13 11:36 -0700

#29048 — Re: subprocess call is not waiting.

On 2012-09-13 16:34, Jean-Michel Pichavant wrote:
> ----- Original Message -----
>> I have a subprocess.call which tries to download a data from a remote
>> server using HTAR. I put the call in a while loop, which tests to
>> see if the download was successful, and if not, loops back around up
>> to five times, just in case my internet connection has a hiccup.
>>
>> Subprocess.call is supposed to wait.
>>
>> But it doesn't work as intended. The loop quickly runs 5 times,
>> starting a new htar command each time. After five times around, my
>> program tells me my download failed, because the target file doesn't
>> yet exist. But it turns out that the download is still
>> happening---five times.
>>
>> When I run htar from the shell, I don't get a shell prompt again
>> until after the download is complete. How come control is returned
>> to python before the htar command is through?
>>
>> I've tried using Popen with wait and/or communicate, but no waiting
>> ever happens. This is troublesome not only because I don't get to
>> post process my data, but because when I run this script for
>> multiple datasets (checking to see whether I have local copies), I
>> quickly get a "Too many open files" error. (I began working on that
>> by trying to use Popopen with fds_close, etc.)
>>
>> Should I just go back to os.system?
>> --
>> http://mail.python.org/mailman/listinfo/python-list
>>
>
> A related subset of code would be useful.
>
> You can use subprocess.PIPE to redirect stdout & stderr et get them with communicate, something like:
>
> proc = subprocess.Popen(['whatever'], stdout=subprocess.PIPE, stdout=subprocess.PIPE)
> stdout, stderr = proc.communicate()
> print stdout
> print stderr
>
> Just by looking at stdout and stderr, you should be able to see why htar is returning so fast.
>
> JM
>
> PS : if you see nothing wrong, is it possible that htar runs asynchronously ?
>
The OP says that it waits when run from the shell.

[toc] | [next] | [standalone]

#29064

Thanks, guys.
MRAB-RedHat 6 64-bit, Python 2.6.5
JM-Here's the relevant stuff from my last try. I've also tried with subprocess.call. Just now I tried shell=True, but it made no difference.

sticking a print(out) in there just prints a blank line in between each iteration. It's not until the 5 trials are finished that I am told: download failed, etc.

from os.path import exists
from subprocess import call
from subprocess import Popen
from shlex import split
from time import sleep

while (exists(file)==0) and (nTries < 5):
   a = Popen(split('htar -xvf ' + htarArgs), stdout=PIPE, stderr=PIPE)
   (out,err) = a.communicate()
   if exists(file)==0:
      nTries += 1
      sleep(0.5)

if exists(file)==0: # now that the file should be moved
   print('download failed: ' + file)
   return 1

I've also tried using shell=True with popopen.

[toc] | [prev] | [next] | [standalone]

#29115

On Thu, Sep 13, 2012 at 11:36 AM,  <paulstaten@gmail.com> wrote:
> Thanks, guys.
> MRAB-RedHat 6 64-bit, Python 2.6.5

In your Unix shell, what does the command:
    type htar
output?

> JM-Here's the relevant stuff from my last try.

If you could give a complete, self-contained example, it would assist
us in troubleshooting your problem.

> I've also tried with subprocess.call. Just now I tried shell=True, but it made no difference.

It's possible that htar uses some trickery to determine whether it's
being invoked from a terminal or by another program, and changes its
behavior accordingly, although I could not find any evidence of that
based on scanning its manpage.

> sticking a print(out) in there just prints a blank line in between each iteration. It's not until the 5 trials are finished that I am told: download failed, etc.
>
> from os.path import exists
> from subprocess import call
> from subprocess import Popen
> from shlex import split
> from time import sleep
>
> while (exists(file)==0) and (nTries < 5):

`file` is the name of a built-in type in Python; it should therefore
not be used as a variable name.
Also, one would normally write that as:
    while not exists(file) and nTries < 5:

>    a = Popen(split('htar -xvf ' + htarArgs), stdout=PIPE, stderr=PIPE)

What's the value of `htarArgs`? (with any sensitive parts anonymized)

Also, you really shouldn't use shlex.split() at run-time like that.
Unless `htarArgs` is already quoted/escaped, you'll get bad results
for many inputs. Use shlex.split() once at the interactive interpreter
to figure out the general form of the tokenization, then use the
static result in your program as a template.

>    (out,err) = a.communicate()
>    if exists(file)==0:
>       nTries += 1
>       sleep(0.5)
>
> if exists(file)==0: # now that the file should be moved
>    print('download failed: ' + file)
>    return 1
>
> I've also tried using shell=True with popopen.

I presume you meant Popen.

Cheers,
Chris

[toc] | [prev] | [next] | [standalone]

#29065

Thanks, guys.
MRAB-RedHat 6 64-bit, Python 2.6.5
JM-Here's the relevant stuff from my last try. I've also tried with subprocess.call. Just now I tried shell=True, but it made no difference.

sticking a print(out) in there just prints a blank line in between each iteration. It's not until the 5 trials are finished that I am told: download failed, etc.

from os.path import exists
from subprocess import call
from subprocess import Popen
from shlex import split
from time import sleep

while (exists(file)==0) and (nTries < 5):
   a = Popen(split('htar -xvf ' + htarArgs), stdout=PIPE, stderr=PIPE)
   (out,err) = a.communicate()
   if exists(file)==0:
      nTries += 1
      sleep(0.5)

if exists(file)==0: # now that the file should be moved
   print('download failed: ' + file)
   return 1

I've also tried using shell=True with popopen.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web