Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #29040 > unrolled thread

subprocess call is not waiting.

Back to article view | Back to comp.lang.python

Contents

  subprocess call is not waiting. paulstaten@gmail.com - 2012-09-13 08:17 -0700
    Re: subprocess call is not waiting. MRAB <python@mrabarnett.plus.com> - 2012-09-13 16:35 +0100
    Re: subprocess call is not waiting. woooee@gmail.com - 2012-09-13 10:24 -0700
      Re: subprocess call is not waiting. Hans Mulder <hansmu@xs4all.nl> - 2012-09-14 10:52 +0200
    Re: subprocess call is not waiting. Chris Rebert <clp2@rebertia.com> - 2012-09-13 22:27 -0700
    Re: subprocess call is not waiting. paulstaten@gmail.com - 2012-09-14 05:22 -0700
      Re: subprocess call is not waiting. Wanderer <wanderer@dialup4less.com> - 2012-09-14 10:38 -0700
      Re: subprocess call is not waiting. Chris Rebert <clp2@rebertia.com> - 2012-09-14 21:02 -0700
        Re: subprocess call is not waiting. paulstaten@gmail.com - 2012-09-15 05:59 -0700
          Re: subprocess call is not waiting. andrea crotti <andrea.crotti.0@gmail.com> - 2012-09-18 14:54 +0100
          Re: subprocess call is not waiting. Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-09-18 15:20 -0400
          Re: subprocess call is not waiting. andrea crotti <andrea.crotti.0@gmail.com> - 2012-09-19 11:26 +0100
            Re: subprocess call is not waiting. Hans Mulder <hansmu@xs4all.nl> - 2012-09-19 15:23 +0200
              Re: subprocess call is not waiting. Gene Heskett <gheskett@wdtv.com> - 2012-09-19 11:57 -0400
              Re: subprocess call is not waiting. andrea crotti <andrea.crotti.0@gmail.com> - 2012-09-19 17:34 +0100
                Re: subprocess call is not waiting. Hans Mulder <hansmu@xs4all.nl> - 2012-09-19 19:31 +0200

#29040 — subprocess call is not waiting.

I have a subprocess.call which tries to download a data from a remote server using HTAR. I put the call in a while loop, which tests to see if the download was successful, and if not, loops back around up to five times, just in case my internet connection has a hiccup.

Subprocess.call is supposed to wait.

But it doesn't work as intended. The loop quickly runs 5 times, starting a new htar command each time. After five times around, my program tells me my download failed, because the target file doesn't yet exist. But it turns out that the download is still happening---five times.

When I run htar from the shell, I don't get a shell prompt again until after the download is complete. How come control is returned to python before the htar command is through?

I've tried using Popen with wait and/or communicate, but no waiting ever happens. This is troublesome not only because I don't get to post process my data, but because when I run this script for multiple datasets (checking to see whether I have local copies), I quickly get a "Too many open files" error. (I began working on that by trying to use Popopen with fds_close, etc.)

Should I just go back to os.system?

[toc] | [next] | [standalone]

#29045

On 2012-09-13 16:17, paulstaten@gmail.com wrote:
> I have a subprocess.call which tries to download a data from a remote server using HTAR. I put the call in a while loop, which tests to see if the download was successful, and if not, loops back around up to five times, just in case my internet connection has a hiccup.
>
> Subprocess.call is supposed to wait.
>
> But it doesn't work as intended. The loop quickly runs 5 times, starting a new htar command each time. After five times around, my program tells me my download failed, because the target file doesn't yet exist. But it turns out that the download is still happening---five times.
>
> When I run htar from the shell, I don't get a shell prompt again until after the download is complete. How come control is returned to python before the htar command is through?
>
> I've tried using Popen with wait and/or communicate, but no waiting ever happens. This is troublesome not only because I don't get to post process my data, but because when I run this script for multiple datasets (checking to see whether I have local copies), I quickly get a "Too many open files" error. (I began working on that by trying to use Popopen with fds_close, etc.)
>
> Should I just go back to os.system?
>
Which OS? Is there some documentation somewhere?

[toc] | [prev] | [next] | [standalone]

#29056

It possibly requires a "shell=True", but without any code on any way to test, we can not say.

[toc] | [prev] | [next] | [standalone]

#29142

On 13/09/12 19:24:46, woooee@gmail.com wrote:
> It possibly requires a "shell=True",

That's almost always a bad idea, and wouldn't affect waiting anyway.

> but without any code or any way to test, we can not say.

That's very true.

-- HansM

[toc] | [prev] | [next] | [standalone]

#29116

On Thu, Sep 13, 2012 at 8:17 AM,  <paulstaten@gmail.com> wrote:
> I have a subprocess.call
<snip>
> But it doesn't work as intended.
<snip>
> Should I just go back to os.system?

Did the os.system() version work?

As of recent Python versions, os.system() is itself implemented using
the `subprocess` module, so if it does work, then it assuredly can be
made to work using the `subprocess` module instead.

Cheers,
Chris

[toc] | [prev] | [next] | [standalone]

#29162

os.system worked fine, and I found something in another section of code that was causing the "Too many open errors." (I was fooled, because output from subprocess call didn't seem to be coming out until the open files error.

I'll go back and play with subprocess.call more, since os.system works. That's interesting about using shlex at run time. Is that just for the sake of computational cost?

[toc] | [prev] | [next] | [standalone]

#29183

On Friday, September 14, 2012 8:22:44 AM UTC-4, pauls...@gmail.com wrote:
> os.system worked fine, and I found something in another section of code that was causing the "Too many open errors." (I was fooled, because output from subprocess call didn't seem to be coming out until the open files error.
> 
> 
> 
> I'll go back and play with subprocess.call more, since os.system works. That's interesting about using shlex at run time. Is that just for the sake of computational cost?

I never got the hang of subprocess, either. I ended up wrapping os.system in a python file and using subprocess to call that with:

subprocess.Popen([sys.executable, 'Wrapper.py'])

This works for me. I'm using Windows 7.

[toc] | [prev] | [next] | [standalone]

#29205

On Fri, Sep 14, 2012 at 5:22 AM,  <paulstaten@gmail.com> wrote:
> os.system worked fine, and I found something in another section of code that was causing the "Too many open errors." (I was fooled, because output from subprocess call didn't seem to be coming out until the open files error.
>
> I'll go back and play with subprocess.call more, since os.system works. That's interesting about using shlex at run time. Is that just for the sake of computational cost?

No, like I said, you'll also get incorrect results. shlex isn't magic.
If the exact command line it's given wouldn't work in the shell, then
it won't magically fix things. Many (most?) dynamic invocations of
shlex.split() are naive and flawed:

>>> import shlex
>>> filename = "my summer vacation.txt"
>>> # the following error is less obvious when the command is more complex
>>> # (and when the filename isn't hardcoded)
>>> cmd = "cat " + filename
>>> shlex.split(cmd)
['cat', 'my', 'summer', 'vacation.txt']
>>> # that's wrong; the entire filename should be a single list element

Equivalent bash error:
chris@mbp ~ $ cat my summer vacation.txt
cat: my: No such file or directory
cat: summer: No such file or directory
cat: vacation.txt: No such file or directory

The right way, in bash:
chris@mbp ~ $ cat my\ summer\ vacation.txt
Last summer, I interned at a tech company and...
chris@mbp ~ $ cat 'my summer vacation.txt'
Last summer, I interned at a tech company and…

And indeed, shlex will get that right too:
>>> shlex.split("cat my\ summer\ vacation.txt")
['cat', 'my summer vacation.txt']
>>> shlex.split("cat 'my summer vacation.txt'")
['cat', 'my summer vacation.txt']

BUT that presumes that your filenames are already pre-quoted or have
had backslashes added, which very seldom is the case in reality. So,
you can either find an escaping function and hope you never forget to
invoke it (cf. SQL injection), or you can figure out the general
tokenization and let `subprocess` handle the rest (cf. prepared
statements):

>>> split('cat examplesimplefilename')
['cat', 'examplesimplefilename']
>>> # Therefore…
>>> def do_cat(filename):
...     cmd = ['cat', filename] # less trivial cases would be more interesting
...     call(cmd)
...
>>> filename = "my summer vacation.txt"
>>> # remember that those quotes are Python literal syntax and aren't in the string itself
>>> print filename
my summer vacation.txt
>>> do_cat(filename)
Last summer, I interned at a tech company and…
>>>

Generally, use (a) deliberately simple test filename(s) with shlex,
then take the resulting list and replace the filename(s) with (a)
variable(s).

Or, just figure out the tokenization without recourse to shlex; it's
not difficult in most cases!
The Note in the Popen docs covers some common tokenization mistakes people make:
http://docs.python.org/library/subprocess.html#subprocess.Popen

Cheers,
Chris

[toc] | [prev] | [next] | [standalone]

#29223

That's a habit I'll make sure to avoid, then.
Thanks, Chris!

[toc] | [prev] | [next] | [standalone]

#29431

I have a similar problem, something which I've never quite understood
about subprocess...
Suppose I do this:

proc = subprocess.Popen(['ls', '-lR'], stdout=subprocess.PIPE,
stderr=subprocess.PIPE)

now I created a process, which has a PID, but it's not running apparently...
It only seems to run when I actually do the wait.

I don't want to make it waiting, so an easy solution is just to use a
thread, but is there a way with subprocess?

[toc] | [prev] | [next] | [standalone]

#29448

On Tue, 18 Sep 2012 14:54:41 +0100, andrea crotti
<andrea.crotti.0@gmail.com> declaimed the following in
gmane.comp.python.general:

> I have a similar problem, something which I've never quite understood
> about subprocess...
> Suppose I do this:
> 
> proc = subprocess.Popen(['ls', '-lR'], stdout=subprocess.PIPE,
> stderr=subprocess.PIPE)
> 
> now I created a process, which has a PID, but it's not running apparently...
> It only seems to run when I actually do the wait.
> 
> I don't want to make it waiting, so an easy solution is just to use a
> thread, but is there a way with subprocess?

	Unless you have a really massive result set from that "ls", that
command probably ran so fast that it is blocked waiting for someone to
read the PIPE.

-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#29478

2012/9/18 Dennis Lee Bieber <wlfraed@ix.netcom.com>:
>
>         Unless you have a really massive result set from that "ls", that
> command probably ran so fast that it is blocked waiting for someone to
> read the PIPE.

I tried also with "ls -lR /" and that definitively takes a while to run,
when I do this:

proc = subprocess.Popen(['ls', '-lR', '/'], stdout=subprocess.PIPE,
stderr=subprocess.PIPE)

nothing is running, only when I actually do
proc.communicate()

I see the process running in top..
Is it still an observation problem?

Anyway I also need to know when the process is over while waiting, so
probably a thread is the only way..

[toc] | [prev] | [next] | [standalone]

#29491

On 19/09/12 12:26:30, andrea crotti wrote:
> 2012/9/18 Dennis Lee Bieber <wlfraed@ix.netcom.com>:
>>
>>         Unless you have a really massive result set from that "ls", that
>> command probably ran so fast that it is blocked waiting for someone to
>> read the PIPE.
> 
> I tried also with "ls -lR /" and that definitively takes a while to run,
> when I do this:
> 
> proc = subprocess.Popen(['ls', '-lR', '/'], stdout=subprocess.PIPE,
> stderr=subprocess.PIPE)
> 
> nothing is running, only when I actually do
> proc.communicate()
> 
> I see the process running in top..
> Is it still an observation problem?

Yes: using "top" is an observation problem.

"Top", as the name suggests, shows only the most active processes.

It's quite possible that your 'ls' process is not active, because
it's waiting for your Python process to read some data from the pipe.

Try using "ps" instead.  Look in thte man page for the correct
options (they differ between platforms).  The default options do
not show all processes, so they may not show the process you're
looking for.

> Anyway I also need to know when the process is over while waiting, so
> probably a thread is the only way..

This sounds confused.

You don't need threads.  When 'ls' finishes, you'll read end-of-file
on the proc.stdout pipe.  You should then call proc.wait() to reap
its exit status (if you don't, you'll leave a zombie process).
Since the process has already finished, the proc.wait() call will
not actually do any waiting.


Hope this helps,

-- HansM

[toc] | [prev] | [next] | [standalone]

#29503

On Wednesday 19 September 2012 11:56:44 Hans Mulder did opine:

> On 19/09/12 12:26:30, andrea crotti wrote:
> > 2012/9/18 Dennis Lee Bieber <wlfraed@ix.netcom.com>:
> >>         Unless you have a really massive result set from that "ls",
> >>         that
> >> 
> >> command probably ran so fast that it is blocked waiting for someone
> >> to read the PIPE.
> > 
> > I tried also with "ls -lR /" and that definitively takes a while to
> > run, when I do this:
> > 
> > proc = subprocess.Popen(['ls', '-lR', '/'], stdout=subprocess.PIPE,
> > stderr=subprocess.PIPE)
> > 
> > nothing is running, only when I actually do
> > proc.communicate()
> > 
> > I see the process running in top..
> > Is it still an observation problem?
> 
> Yes: using "top" is an observation problem.
> 
> "Top", as the name suggests, shows only the most active processes.
> 
Which is why I run htop in a shell 100% of the time.  With htop, you can 
scroll down and see everything.

> It's quite possible that your 'ls' process is not active, because
> it's waiting for your Python process to read some data from the pipe.
> 
> Try using "ps" instead.  Look in thte man page for the correct
> options (they differ between platforms).  The default options do
> not show all processes, so they may not show the process you're
> looking for.
> 
> > Anyway I also need to know when the process is over while waiting, so
> > probably a thread is the only way..
> 
> This sounds confused.
> 
> You don't need threads.  When 'ls' finishes, you'll read end-of-file
> on the proc.stdout pipe.  You should then call proc.wait() to reap
> its exit status (if you don't, you'll leave a zombie process).
> Since the process has already finished, the proc.wait() call will
> not actually do any waiting.
> 
> 
> Hope this helps,
> 
> -- HansM


Cheers, Gene
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
My web page: <http://coyoteden.dyndns-free.com:85/gene> is up!
To know Edina is to reject it.
		-- Dudley Riggs, "The Year the Grinch Stole the Election"

[toc] | [prev] | [next] | [standalone]

#29506

2012/9/19 Hans Mulder <hansmu@xs4all.nl>:
> Yes: using "top" is an observation problem.
>
> "Top", as the name suggests, shows only the most active processes.

Sure but "ls -lR /" is a very active process if you try to run it..
Anyway as written below I don't need this anymore.

>
> It's quite possible that your 'ls' process is not active, because
> it's waiting for your Python process to read some data from the pipe.
>
> Try using "ps" instead.  Look in thte man page for the correct
> options (they differ between platforms).  The default options do
> not show all processes, so they may not show the process you're
> looking for.
>
>> Anyway I also need to know when the process is over while waiting, so
>> probably a thread is the only way..
>
> This sounds confused.
>
> You don't need threads.  When 'ls' finishes, you'll read end-of-file
> on the proc.stdout pipe.  You should then call proc.wait() to reap
> its exit status (if you don't, you'll leave a zombie process).
> Since the process has already finished, the proc.wait() call will
> not actually do any waiting.
>
>
> Hope this helps,
>


Well there is a process which has to do two things, monitor
periodically some external conditions (filesystem / db), and launch a
process that can take very long time.

So I can't put a wait anywhere, or I'll stop everything else.  But at
the same time I need to know when the process is finished, which I
could do but without a wait might get hacky.

So I'm quite sure I just need to run the subprocess in a subthread
unless I'm missing something obvious..

[toc] | [prev] | [next] | [standalone]

#29509

On 19/09/12 18:34:58, andrea crotti wrote:
> 2012/9/19 Hans Mulder <hansmu@xs4all.nl>:
>> Yes: using "top" is an observation problem.
>>
>> "Top", as the name suggests, shows only the most active processes.
> 
> Sure but "ls -lR /" is a very active process if you try to run it..

Not necessarily:

>> It's quite possible that your 'ls' process is not active because
>> it's waiting for your Python process to read some data from the pipe.

Hope this helps,

-- HansM

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web