Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder2.enfer-du-nord.net!feeder.news-service.com!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
Sender: kushal.kumaran@gmail.com
In-Reply-To: <CAFB6qZuFNbK80QO6=kdzLZY2RV1w=q35wTzQdxZxJN33iYPg-w@mail.gmail.com>
References: <CAFB6qZuFNbK80QO6=kdzLZY2RV1w=q35wTzQdxZxJN33iYPg-w@mail.gmail.com>
From: Kushal Kumaran <kushal.kumaran+python@gmail.com>
Date: Sat, 10 Sep 2011 14:49:08 +0530
Subject: Re: A bit of a boggle about subprocess.poll() and the codes it receives from a process
To: J <dreadpiratejeff@gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Cc: Python List <python-list@python.org>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.929.1315646370.27778.python-list@python.org>
Lines: 122
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:13056

On Fri, Sep 9, 2011 at 11:02 PM, J <dreadpiratejeff@gmail.com> wrote:
> Hi,
> I need a bit of help sorting this out...
> I have a memory test script that is a bit of compiled C. =C2=A0The test i=
tself
> can only ever return a 0 or 1 exit code, this is explicitly coded and the=
re
> are no other options.
> I also have a wrapper test script that calls the C program that should al=
so
> only return 0 or 1 on completion.
> The problem i'm encountering, however, involves the return code when
> subprocess.poll() is called against the running memory test process. =C2=
=A0The
> current code in my wrapper program looks like this:
> def run_processes(self, number, command):
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 passed =3D True
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 pipe =3D []
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 for i in range(number):
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pipe.append(self._command(comma=
nd))
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 print "Started: process %u pid =
%u: %s" % (i, pipe[i].pid,
> command)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 sys.stdout.flush()
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 waiting =3D True
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 while waiting:
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 waiting =3D False
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 for i in range(number):
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if pipe[i]:
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 lin=
e =3D pipe[i].communicate()[0]
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if =
line and len(line) > 1:
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 print "process %u pid %u: %s" % (i, pipe[i].pid,
> line)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 sys.stdout.flush()
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if =
pipe[i].poll() =3D=3D -1:
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 waiting =3D True
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 els=
e:
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 return_value =3D pipe[i].poll()
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 if return_value !=3D 0:
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 print "Error: process =C2=A0%u pid %u retuned %u" =
%
> (i, pipe[i].pid, return_value)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 passed =3D False
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 print "process %u pid %u returned success" % (i,
> pipe[i].pid)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 pipe[i] =3D None
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 sys.stdout.flush()
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 return passed
> So what happens here is that in the waiting loop, if pipe[i].poll returns=
 a
> -1, we keep waiting, and then if it returns anything OTHER than -1, we ex=
it
> and return the return code.

Does self._command return a subprocess.Popen object?  The
documentation at
http://docs.python.org/library/subprocess.html#subprocess.Popen.poll
says that the poll method sets and returns the returncode attribute.
returncode is expected to be None until the process terminates, and a
negative value indicates that the subprocess has been terminated
because of a signal.  If poll() returns -1, it means that the process
has been terminated by signal number 1 (probably SIGHUP).

> BUT, I'm getting, in some cases, a return code of 127, which is impossibl=
e
> to get from the memory test program.
> The output from this bit of code looks like this in a failing situation:
> Error: process 0 pid 2187 retuned 127
> process 0 pid 2187 returned success
> Error: process 1 pid 2188 retuned 127
> process 1 pid 2188 returned success
> I'm thinking that I'm hitting some sort of race here where the kernel is
> reporting -1 while the process is running, then returns 127 or some other
> status when the process is being killed and then finally 0 or 1 after the
> process has completely closed out. =C2=A0I "think" that the poll picks up=
 this
> intermediate exit status and immediately exits the loop, instead of waiti=
ng
> for a 0 or 1.
> I've got a modified version that I'm getting someone to test for me now t=
hat
> changes
> =C2=A0if pipe[i].poll() =3D=3D -1:
> =C2=A0 =C2=A0 =C2=A0waiting =3D True
> to this
> if pipe[i].poll() not in [0,1]:
> =C2=A0 =C2=A0 waiting =3D True
> So my real question is: am I on the right track here, and am I correct in=
 my
> guess that the kernel is reporting different status codes to
> subprocess.poll() during the shutdown of the polled process?
>

I'm unaware of any such race condition.  It is more likely that the
program you are running can return such error codes.  Perhaps you
should examine its stderr output.

It's unclear what you're trying to do here.  If you'd just like to
wait until all the started processes are finished, you should use the
wait() method instead of the poll() method.  You should also note that
the communicate() method already waits for the process to complete, so
further calls to poll() and wait() are superfluous.  Normally, after
communicate() returns, you would simply check pipe[i].returncode and
be on your way.

--=20
regards,
kushal