Path: csiph.com!usenet.pasdenom.info!news.albasani.net!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!194.109.133.85.MISMATCH!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
Sender: chris@rebertia.com
In-Reply-To: <0b598d7f-7137-47cc-8eed-80ab450536c5@googlegroups.com>
References: <d49e1e71-afe2-4aab-8c67-3e2eea6533a6@googlegroups.com> <0b598d7f-7137-47cc-8eed-80ab450536c5@googlegroups.com>
Date: Fri, 14 Sep 2012 21:02:22 -0700
Subject: Re: subprocess call is not waiting.
From: Chris Rebert <clp2@rebertia.com>
To: paulstaten@gmail.com
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Cc: python-list@python.org
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.734.1347681745.27098.python-list@python.org>
Lines: 78
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:29205

On Fri, Sep 14, 2012 at 5:22 AM,  <paulstaten@gmail.com> wrote:
> os.system worked fine, and I found something in another section of code t=
hat was causing the "Too many open errors." (I was fooled, because output f=
rom subprocess call didn't seem to be coming out until the open files error=
.
>
> I'll go back and play with subprocess.call more, since os.system works. T=
hat's interesting about using shlex at run time. Is that just for the sake =
of computational cost?

No, like I said, you'll also get incorrect results. shlex isn't magic.
If the exact command line it's given wouldn't work in the shell, then
it won't magically fix things. Many (most?) dynamic invocations of
shlex.split() are naive and flawed:

>>> import shlex
>>> filename =3D "my summer vacation.txt"
>>> # the following error is less obvious when the command is more complex
>>> # (and when the filename isn't hardcoded)
>>> cmd =3D "cat " + filename
>>> shlex.split(cmd)
['cat', 'my', 'summer', 'vacation.txt']
>>> # that's wrong; the entire filename should be a single list element

Equivalent bash error:
chris@mbp ~ $ cat my summer vacation.txt
cat: my: No such file or directory
cat: summer: No such file or directory
cat: vacation.txt: No such file or directory

The right way, in bash:
chris@mbp ~ $ cat my\ summer\ vacation.txt
Last summer, I interned at a tech company and...
chris@mbp ~ $ cat 'my summer vacation.txt'
Last summer, I interned at a tech company and=E2=80=A6

And indeed, shlex will get that right too:
>>> shlex.split("cat my\ summer\ vacation.txt")
['cat', 'my summer vacation.txt']
>>> shlex.split("cat 'my summer vacation.txt'")
['cat', 'my summer vacation.txt']

BUT that presumes that your filenames are already pre-quoted or have
had backslashes added, which very seldom is the case in reality. So,
you can either find an escaping function and hope you never forget to
invoke it (cf. SQL injection), or you can figure out the general
tokenization and let `subprocess` handle the rest (cf. prepared
statements):

>>> split('cat examplesimplefilename')
['cat', 'examplesimplefilename']
>>> # Therefore=E2=80=A6
>>> def do_cat(filename):
...     cmd =3D ['cat', filename] # less trivial cases would be more intere=
sting
...     call(cmd)
...
>>> filename =3D "my summer vacation.txt"
>>> # remember that those quotes are Python literal syntax and aren't in th=
e string itself
>>> print filename
my summer vacation.txt
>>> do_cat(filename)
Last summer, I interned at a tech company and=E2=80=A6
>>>

Generally, use (a) deliberately simple test filename(s) with shlex,
then take the resulting list and replace the filename(s) with (a)
variable(s).

Or, just figure out the tokenization without recourse to shlex; it's
not difficult in most cases!
The Note in the Popen docs covers some common tokenization mistakes people =
make:
http://docs.python.org/library/subprocess.html#subprocess.Popen

Cheers,
Chris