Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #111537 > unrolled thread

Passing File Descriptors To Subprocesses

Back to article view | Back to comp.lang.python

Contents

  Passing File Descriptors To Subprocesses Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-07-16 18:59 -0700
    Re: Passing File Descriptors To Subprocesses eryk sun <eryksun@gmail.com> - 2016-07-17 03:00 +0000
      Re: Passing File Descriptors To Subprocesses Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-07-17 00:06 -0700

#111537 — Passing File Descriptors To Subprocesses

A few years ago I wrote a tool <https://github.com/ldo/apitizer> to do comparisons between different versions of the Android API. Then one day, after a Python update (from the dates, it had to have been some version of 3.2), it stopped working. It took quite a while--over a year--until I figured out what was happening, and I could move my code from Python 2 back to Python 3 again.

The various subprocess functions <https://docs.python.org/3/library/subprocess.html> have arguments called “close_fds” and “pass_fds”, which specify which file descriptors are to be left open in the child process. Yet no matter what I set these to, it seemed I could not pass my pipes to a subprocess.

What the docs *don’t* tell you is that these arguments do not control what happens after the exec. The file descriptors that are kept open are only those which do not have the FD_CLOEXEC flags set in their fcntl settings.

Remember how the subprocess functions work:
* first, a fork(2) call is executed, then
* the child process does execve(2) (or some convenience variant of this) to actually execute the command or program that you specified.

The “close_fds” and “pass_fds” args only matter in the first step. You need to specify these, *and* have the right CLOEXEC settings on those file descriptors for the second step. Leave out either one, and your child process does not get the file descriptors.

Python 3.4 has added special calls <https://docs.python.org/3/library/os.html#inheritance-of-file-descriptors> to manage this CLOEXEC setting. My code uses fcntl <https://docs.python.org/3/library/fcntl.html>.

The docs for the subprocess module need to make this requirement clear.

[toc] | [next] | [standalone]

#111541

On Sun, Jul 17, 2016 at 1:59 AM, Lawrence D’Oliveiro
<lawrencedo99@gmail.com> wrote:
> The various subprocess functions <https://docs.python.org/3/library/subprocess.html> have
> arguments called “close_fds” and “pass_fds”, which specify which file descriptors are to be
> left open in the child process. Yet no matter what I set these to, it seemed I could not pass
> my pipes to a subprocess.
>
> What the docs *don’t* tell you is that these arguments do not control what happens after the
> exec. The file descriptors that are kept open are only those which do not have the
> FD_CLOEXEC flags set in their fcntl settings.

It works correctly in 3.4+, which makes the pass_fds file descriptors
inheritable in the child, after fork. See issue 18571 and PEP 446,
section "Other Changes":

http://bugs.python.org/issue18571
https://www.python.org/dev/peps/pep-0446/#other-changes

For example:

    Python 3.5.1+ (default, Mar 30 2016, 22:46:26)
    [GCC 5.3.1 20160330] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import os, subprocess
    >>> fdr, fdw = os.pipe()
    >>> fdw
    4
    >>> os.get_inheritable(fdw)
    False
    >>> subprocess.call(['python3'], pass_fds=[fdw])

child:

    Python 3.5.1+ (default, Mar 30 2016, 22:46:26)
    [GCC 5.3.1 20160330] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import os
    >>> os.write(4, b'spam')
    4
    >>> exit()

parent:

    0
    >>> os.read(fdr, 4)
    b'spam'

[toc] | [prev] | [next] | [standalone]

#111550

On Sunday, July 17, 2016 at 3:01:36 PM UTC+12, eryk sun wrote:

> It works correctly in 3.4+ ...

Yup, confirmed it happens only in 3.3.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web