Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #91839 > unrolled thread

Re: fork/exec & close file descriptors

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: fork/exec & close file descriptors Skip Montanaro <skip.montanaro@gmail.com> - 2015-06-02 08:58 -0500
    Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-02 18:28 +0300
      Re: fork/exec & close file descriptors Skip Montanaro <skip.montanaro@gmail.com> - 2015-06-02 10:42 -0500
        Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-02 19:24 +0300
          Re: fork/exec & close file descriptors Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2015-06-02 16:37 +0000
      Re: fork/exec & close file descriptors MrJean1 <MrJean1@gmail.com> - 2015-06-12 13:43 -0700
    Re: fork/exec & close file descriptors Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid> - 2015-06-02 17:59 +0200
      Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-02 23:05 +0300
        Re: fork/exec & close file descriptors Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid> - 2015-06-02 23:06 +0200
          Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-03 01:24 +0300
            Re: fork/exec & close file descriptors Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid> - 2015-06-03 09:21 +0200
              Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-03 10:41 +0300
                Re: fork/exec & close file descriptors alister <alister.nospam.ware@ntlworld.com> - 2015-06-03 09:38 +0000
                  Re: fork/exec & close file descriptors Steven D'Aprano <steve@pearwood.info> - 2015-06-03 22:07 +1000
                    Re: fork/exec & close file descriptors alister <alister.nospam.ware@ntlworld.com> - 2015-06-03 12:18 +0000
                      Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-03 15:27 +0300
                        Re: fork/exec & close file descriptors alister <alister.nospam.ware@ntlworld.com> - 2015-06-03 13:05 +0000
                          Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-03 17:49 +0300
                    Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-03 15:25 +0300
                      Re: fork/exec & close file descriptors random832@fastmail.us - 2015-06-03 08:54 -0400
                        Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-03 16:08 +0300
                          Re: fork/exec & close file descriptors random832@fastmail.us - 2015-06-03 09:21 -0400
                          Re: fork/exec & close file descriptors Chris Angelico <rosuav@gmail.com> - 2015-06-03 23:32 +1000
                          Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-03 16:33 +0300
                            Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-03 17:43 +0300
                              Re: fork/exec & close file descriptors random832@fastmail.us - 2015-06-03 16:08 -0400
                                Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-04 00:09 +0300
                          Re: fork/exec & close file descriptors random832@fastmail.us - 2015-06-03 16:07 -0400
                          Re: fork/exec & close file descriptors Chris Angelico <rosuav@gmail.com> - 2015-06-04 07:54 +1000
          Re: fork/exec & close file descriptors Chris Angelico <rosuav@gmail.com> - 2015-06-03 09:54 +1000
            Re: fork/exec & close file descriptors Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid> - 2015-06-03 09:11 +0200
              Re: fork/exec & close file descriptors random832@fastmail.us - 2015-06-03 08:21 -0400
                Re: fork/exec & close file descriptors Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid> - 2015-06-03 15:16 +0200
          Re: fork/exec & close file descriptors Skip Montanaro <skip.montanaro@gmail.com> - 2015-06-02 20:21 -0500

Page 2 of 2 — ← Prev page 1 [2]

#91957

random832@fastmail.us:

> Why does the child process need to report the error at all? The parent
> process will find out naturally when *it* tries to close the same file
> descriptor.

That's not how it goes.

File descriptors are reference counted in the Linux kernel. Closes are
no-ops except for the last one that brings the reference count to zero.

If the parent should close the file before the child, no error is
returned to the parent.


Marko

[toc] | [prev] | [next] | [standalone]

#91961

On Wed, Jun 3, 2015, at 09:08, Marko Rauhamaa wrote:
> random832@fastmail.us:
> 
> > Why does the child process need to report the error at all? The parent
> > process will find out naturally when *it* tries to close the same file
> > descriptor.
> 
> That's not how it goes.
> 
> File descriptors are reference counted in the Linux kernel. Closes are
> no-ops except for the last one that brings the reference count to zero.
> 
> If the parent should close the file before the child, no error is
> returned to the parent.

Why would the parent close it before the child? Your scenario doesn't
seem to have anything to do with how people actually use subprocesses.

[toc] | [prev] | [next] | [standalone]

#91963

On Wed, Jun 3, 2015 at 11:21 PM,  <random832@fastmail.us> wrote:
> On Wed, Jun 3, 2015, at 09:08, Marko Rauhamaa wrote:
>> random832@fastmail.us:
>>
>> > Why does the child process need to report the error at all? The parent
>> > process will find out naturally when *it* tries to close the same file
>> > descriptor.
>>
>> That's not how it goes.
>>
>> File descriptors are reference counted in the Linux kernel. Closes are
>> no-ops except for the last one that brings the reference count to zero.
>>
>> If the parent should close the file before the child, no error is
>> returned to the parent.
>
> Why would the parent close it before the child? Your scenario doesn't
> seem to have anything to do with how people actually use subprocesses.

Write an editor that opens a file and holds it open until the user's
done with it. Have something that lets you shell out for whatever
reason. Then trigger the shell-out, and instantly SIGSTOP the child
process, before it does its work - or just have a really heavily
loaded system, so it can't get a time slice. Now close the file in the
UI, which results in the file being closed in the parent. Right, now
let the child run... and there it goes, closing the file.

Mightn't be a common situation (given that the amount of code executed
between forking and closing FDs is not going to be much, and isn't
going to involve reading from disk or anything), but it can certainly
happen.

ChrisA

[toc] | [prev] | [next] | [standalone]

#91964

Marko Rauhamaa <marko@pacujo.net>:

> random832@fastmail.us:
>
>> Why does the child process need to report the error at all? The parent
>> process will find out naturally when *it* tries to close the same file
>> descriptor.
>
> That's not how it goes.
>
> File descriptors are reference counted in the Linux kernel. Closes are
> no-ops except for the last one that brings the reference count to zero.
>
> If the parent should close the file before the child, no error is
> returned to the parent.

First of all, it's not the descriptors that are refcounted -- it's the
files referred to by the descriptors.

However, I was also wrong about close() being a no-operation. Here's the
relevant kernel source code snippet:

========================================================================
int __close_fd(struct files_struct *files, unsigned fd)
{
	struct file *file;
	struct fdtable *fdt;

	spin_lock(&files->file_lock);
	fdt = files_fdtable(files);
	if (fd >= fdt->max_fds)
		goto out_unlock;
	file = fdt->fd[fd];
	if (!file)
		goto out_unlock;
	rcu_assign_pointer(fdt->fd[fd], NULL);
	__clear_close_on_exec(fd, fdt);
	__put_unused_fd(files, fd);
	spin_unlock(&files->file_lock);
	return filp_close(file, files);

out_unlock:
	spin_unlock(&files->file_lock);
	return -EBADF;
}

int filp_close(struct file *filp, fl_owner_t id)
{
	int retval = 0;

	if (!file_count(filp)) {
		printk(KERN_ERR "VFS: Close: file count is 0\n");
		return 0;
	}

	if (filp->f_op->flush)
		retval = filp->f_op->flush(filp, id);

	if (likely(!(filp->f_mode & FMODE_PATH))) {
		dnotify_flush(filp, id);
		locks_remove_posix(filp, id);
	}
	fput(filp);
	return retval;
}
========================================================================

What is revealed is that:

 1. The file descriptor is released regardless of the return value of
    close(2):

    __put_unused_fd(files, fd);

 2. The file object's refcount is decremented accordingly regardless of
    the return value of close(2).

 3. The return value reflects the success of the optional flush() method
    of the file object.

 4. The flush() method is called with each call to close(2), not only
    the last one.

IOW, os.close() closes the file even if it should report a failure. I
couldn't have guessed that behavior from the man page.

So the strategy you proposed is the right one: have the child process
ignore any possible errors from os.close(). The parent will have an
opportunity to deal with them.

And now Linux is back in the good graces, only the man page is
misleading.


Marko

[toc] | [prev] | [next] | [standalone]

#91969

Marko Rauhamaa <marko@pacujo.net>:

> So the strategy you proposed is the right one: have the child process
> ignore any possible errors from os.close(). The parent will have an
> opportunity to deal with them.
>
> And now Linux is back in the good graces, only the man page is
> misleading.

However, the child process needs to be prepared for os.close() to block
indefinitely because of an NFS problem or because SO_LINGER has been
specified by the parent, for example. Setting the close-on-exec flag
doesn't help there.


Marko

[toc] | [prev] | [next] | [standalone]

#91990

On Wed, Jun 3, 2015, at 10:43, Marko Rauhamaa wrote:
> However, the child process needs to be prepared for os.close() to block
> indefinitely because of an NFS problem or because SO_LINGER has been
> specified by the parent, for example. Setting the close-on-exec flag
> doesn't help there.

Out of curiosity, does exec block in this situation?

[toc] | [prev] | [next] | [standalone]

#91997

random832@fastmail.us:

> On Wed, Jun 3, 2015, at 10:43, Marko Rauhamaa wrote:
>> However, the child process needs to be prepared for os.close() to
>> block indefinitely because of an NFS problem or because SO_LINGER has
>> been specified by the parent, for example. Setting the close-on-exec
>> flag doesn't help there.
>
> Out of curiosity, does exec block in this situation?

I didn't try it, but it is apparent in the source code:

========================================================================
void do_close_on_exec(struct files_struct *files)
{
	unsigned i;
	struct fdtable *fdt;

	/* exec unshares first */
	spin_lock(&files->file_lock);
	for (i = 0; ; i++) {
		unsigned long set;
		unsigned fd = i * BITS_PER_LONG;
		fdt = files_fdtable(files);
		if (fd >= fdt->max_fds)
			break;
		set = fdt->close_on_exec[i];
		if (!set)
			continue;
		fdt->close_on_exec[i] = 0;
		for ( ; set ; fd++, set >>= 1) {
			struct file *file;
			if (!(set & 1))
				continue;
			file = fdt->fd[fd];
			if (!file)
				continue;
			rcu_assign_pointer(fdt->fd[fd], NULL);
			__put_unused_fd(files, fd);
			spin_unlock(&files->file_lock);
			filp_close(file, files);
			cond_resched();
			spin_lock(&files->file_lock);
		}

	}
	spin_unlock(&files->file_lock);
}

int filp_close(struct file *filp, fl_owner_t id)
{
	int retval = 0;

	if (!file_count(filp)) {
		printk(KERN_ERR "VFS: Close: file count is 0\n");
		return 0;
	}

	if (filp->f_op->flush)
		retval = filp->f_op->flush(filp, id);

	if (likely(!(filp->f_mode & FMODE_PATH))) {
		dnotify_flush(filp, id);
		locks_remove_posix(filp, id);
	}
	fput(filp);
	return retval;
}
========================================================================

Now, the kernel NFS code specifies a flush() method, which can block and
even fail.

Sockets don't have a flush() method. So closing a socket cannot fail.
However, fput(), which decrements the reference count, may block if
lingering has been specified for the socket.

So I wasn't all that wrong earlier after all: whoever closes a socket
last will linger. Thus, the parent (who wanted to linger) might zip
through closing a socket while the unwitting child process will suffer
the lingering delay before it gets to exec.

However, there's this comment in inet_release():

		/* [...]
		 * If the close is due to the process exiting, we never
		 * linger..
		 */


Marko

[toc] | [prev] | [next] | [standalone]

#91989

On Wed, Jun 3, 2015, at 09:32, Chris Angelico wrote:
> Write an editor that opens a file and holds it open until the user's
> done with it. Have something that lets you shell out for whatever
> reason. Then trigger the shell-out, and instantly SIGSTOP the child
> process, before it does its work - or just have a really heavily
> loaded system, so it can't get a time slice. Now close the file in the
> UI, which results in the file being closed in the parent. Right, now
> let the child run... and there it goes, closing the file.

The parent should be waiting for the child process. If it shouldn't wait
for the command, then the child process should spawn a grandchild
process, after closing the file descriptors. This is how the text editor
I use actually works (more or less. In fact, the way to run a process it
won't wait for is to run it as a background shell command. If you STOP
the shell itself, the editor will be stuck waiting.)

[toc] | [prev] | [next] | [standalone]

#92007

On Thu, Jun 4, 2015 at 6:07 AM,  <random832@fastmail.us> wrote:
> On Wed, Jun 3, 2015, at 09:32, Chris Angelico wrote:
>> Write an editor that opens a file and holds it open until the user's
>> done with it. Have something that lets you shell out for whatever
>> reason. Then trigger the shell-out, and instantly SIGSTOP the child
>> process, before it does its work - or just have a really heavily
>> loaded system, so it can't get a time slice. Now close the file in the
>> UI, which results in the file being closed in the parent. Right, now
>> let the child run... and there it goes, closing the file.
>
> The parent should be waiting for the child process. If it shouldn't wait
> for the command, then the child process should spawn a grandchild
> process, after closing the file descriptors. This is how the text editor
> I use actually works (more or less. In fact, the way to run a process it
> won't wait for is to run it as a background shell command. If you STOP
> the shell itself, the editor will be stuck waiting.)

Really? I thought forking, execing, and not immediately waiting, was a
standard way to trigger an asynchronous action. My editor lets me
start something and then keep working in the editor, and see the
output from the command when it's ready. (Simple example: A "git push"
might take a long time if the network's slow, and I want to know if it
errors out, but most likely I'll just see the expected messages come
up and that's that.) The parent definitely shouldn't immediately wait
on the child; and it shouldn't disown the child via a second forking
because it wants to report on the child's completion. So no, I don't
think insta-waiting is right in all situations.

ChrisA

[toc] | [prev] | [next] | [standalone]

#91912

On Wed, Jun 3, 2015 at 7:06 AM, Alain Ketterlin
<alain@universite-de-strasbourg.fr.invalid> wrote:
> I've no idea what the OP's program was doing, so I'm not going to split
> hairs. I can't imagine why one would like to mass-close an arbitrary set
> of file descriptors, and I think APIs like os.closerange() are toxic and
> an appeal to sloppy programming.

When you fork, you get a duplicate referent to every open file in both
parent and child. Closing them all in the child is very common, as it
allows the parent to continue owning those file descriptors (so that
when you close it in the parent, the resource is really closed). One
notable example is with listening sockets; bind/listen in the parent,
then fork (maybe to handle a client), then terminate the parent
process. You now cannot restart the parent without aborting the child,
as the child now owns that listening socket (even if it never wants to
use it). There are some specific ways around this, but not on all OSes
(eg Linux only added support for SO_REUSEPORT in 3.9), and the best
way has always been to make sure the children don't hang onto the
listening socket. (There are other good reasons for doing this, too.)

ChrisA

[toc] | [prev] | [next] | [standalone]

#91924

Chris Angelico <rosuav@gmail.com> writes:

> On Wed, Jun 3, 2015 at 7:06 AM, Alain Ketterlin
> <alain@universite-de-strasbourg.fr.invalid> wrote:
>> I've no idea what the OP's program was doing, so I'm not going to split
>> hairs. I can't imagine why one would like to mass-close an arbitrary set
>> of file descriptors, and I think APIs like os.closerange() are toxic and
>> an appeal to sloppy programming.
>
> When you fork, you get a duplicate referent to every open file in both
> parent and child. [...]

Thank you, I know this. What I mean is: what are the reasons that you
cannot access your file descriptors one by one? To me closing a range of
descriptors has absolutely no meaning, simply because ranges have no
meaning for file descriptors (they're not ordered in any way). What if
some library uses its own descriptors that happen to lie in your
"range"? Etc.

-- Alain.

[toc] | [prev] | [next] | [standalone]

#91950

On Wed, Jun 3, 2015, at 03:11, Alain Ketterlin wrote:
> Thank you, I know this. What I mean is: what are the reasons that you
> cannot access your file descriptors one by one? To me closing a range of
> descriptors has absolutely no meaning, simply because ranges have no
> meaning for file descriptors (they're not ordered in any way). What if
> some library uses its own descriptors that happen to lie in your
> "range"? Etc.

The context in which this is useful is that you've just forked, and
you're about to exec. "Some library" isn't going to ever get back
control within the current process. Generally the range of file
descriptors you want to close is (e.g.) 3-Infinity, after you've already
got 0 1 and 2 pointing to where you want them (whatever redirected file
or pipe).

[toc] | [prev] | [next] | [standalone]

#91959

random832@fastmail.us writes:

> On Wed, Jun 3, 2015, at 03:11, Alain Ketterlin wrote:
>> Thank you, I know this. What I mean is: what are the reasons that you
>> cannot access your file descriptors one by one? To me closing a range of
>> descriptors has absolutely no meaning, simply because ranges have no
>> meaning for file descriptors (they're not ordered in any way). What if
>> some library uses its own descriptors that happen to lie in your
>> "range"? Etc.
>
> The context in which this is useful is that you've just forked, and
> you're about to exec. "Some library" isn't going to ever get back
> control within the current process.

Any decent library will (on linux) use close-on-exec, or even have
fini/dtor functions to clean up. Any that does not is buggy. But of
course, if you shoot in their feet...

> Generally the range of file descriptors you want to close is (e.g.)
> 3-Infinity, after you've already got 0 1 and 2 pointing to where you
> want them (whatever redirected file or pipe).

Closing 3-... is meaningless, probably useless, and potentially harmful.

-- Alain.

[toc] | [prev] | [next] | [standalone]

#91916

Folks, I'm not interested in rehashing this. I don't know what all the
open file descriptors are. Some/many/most will have been opened by
underlying C++ libraries and will not have been opened by code at the
Python level. I do think the addition of an os.fsync() attempt on each
file descriptor before calling os.close() makes some sense to me.
Beyond that, I'm not sure there's more that can be done.

I sent my last message simply because I was surprised os.closerange()
existed. I have no idea why it didn't turn up in my original
searching.

Skip

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.python

csiph-web