Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #91839 > unrolled thread

Re: fork/exec & close file descriptors

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: fork/exec & close file descriptors Skip Montanaro <skip.montanaro@gmail.com> - 2015-06-02 08:58 -0500
    Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-02 18:28 +0300
      Re: fork/exec & close file descriptors Skip Montanaro <skip.montanaro@gmail.com> - 2015-06-02 10:42 -0500
        Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-02 19:24 +0300
          Re: fork/exec & close file descriptors Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2015-06-02 16:37 +0000
      Re: fork/exec & close file descriptors MrJean1 <MrJean1@gmail.com> - 2015-06-12 13:43 -0700
    Re: fork/exec & close file descriptors Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid> - 2015-06-02 17:59 +0200
      Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-02 23:05 +0300
        Re: fork/exec & close file descriptors Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid> - 2015-06-02 23:06 +0200
          Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-03 01:24 +0300
            Re: fork/exec & close file descriptors Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid> - 2015-06-03 09:21 +0200
              Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-03 10:41 +0300
                Re: fork/exec & close file descriptors alister <alister.nospam.ware@ntlworld.com> - 2015-06-03 09:38 +0000
                  Re: fork/exec & close file descriptors Steven D'Aprano <steve@pearwood.info> - 2015-06-03 22:07 +1000
                    Re: fork/exec & close file descriptors alister <alister.nospam.ware@ntlworld.com> - 2015-06-03 12:18 +0000
                      Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-03 15:27 +0300
                        Re: fork/exec & close file descriptors alister <alister.nospam.ware@ntlworld.com> - 2015-06-03 13:05 +0000
                          Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-03 17:49 +0300
                    Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-03 15:25 +0300
                      Re: fork/exec & close file descriptors random832@fastmail.us - 2015-06-03 08:54 -0400
                        Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-03 16:08 +0300
                          Re: fork/exec & close file descriptors random832@fastmail.us - 2015-06-03 09:21 -0400
                          Re: fork/exec & close file descriptors Chris Angelico <rosuav@gmail.com> - 2015-06-03 23:32 +1000
                          Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-03 16:33 +0300
                            Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-03 17:43 +0300
                              Re: fork/exec & close file descriptors random832@fastmail.us - 2015-06-03 16:08 -0400
                                Re: fork/exec & close file descriptors Marko Rauhamaa <marko@pacujo.net> - 2015-06-04 00:09 +0300
                          Re: fork/exec & close file descriptors random832@fastmail.us - 2015-06-03 16:07 -0400
                          Re: fork/exec & close file descriptors Chris Angelico <rosuav@gmail.com> - 2015-06-04 07:54 +1000
          Re: fork/exec & close file descriptors Chris Angelico <rosuav@gmail.com> - 2015-06-03 09:54 +1000
            Re: fork/exec & close file descriptors Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid> - 2015-06-03 09:11 +0200
              Re: fork/exec & close file descriptors random832@fastmail.us - 2015-06-03 08:21 -0400
                Re: fork/exec & close file descriptors Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid> - 2015-06-03 15:16 +0200
          Re: fork/exec & close file descriptors Skip Montanaro <skip.montanaro@gmail.com> - 2015-06-02 20:21 -0500

Page 1 of 2  [1] 2  Next page →

#91839 — Re: fork/exec & close file descriptors

Reviving (and concluding) a thread I started a couple weeks ago, I asked:

> The basic fork/exec dance is not a problem, but how do I discover
> all the open file descriptors in the new child process to make sure
> they get closed? Do I simply start at fd 3 and call os.close() on
> everything up to some largish fd number?

I wanted this again today (for different reasons than before).
Googling for "python close all file descriptors" returned the os
module docs as the first hit, and lo and behold, what do I see
documented? os.closerange (new in 2.6):

    os.closerange(fd_low, fd_high)
    Close all file descriptors from fd_low (inclusive) to fd_high
    (exclusive), ignoring errors.

Guido's time machine strikes again...

Skip

[toc] | [next] | [standalone]

#91850

Skip Montanaro <skip.montanaro@gmail.com>:

>     os.closerange(fd_low, fd_high)
>     Close all file descriptors from fd_low (inclusive) to fd_high
>     (exclusive), ignoring errors.
>
> Guido's time machine strikes again...

The only problem is that you don't know how high you need to go in
general.


Marko

[toc] | [prev] | [next] | [standalone]

#91851

On Tue, Jun 2, 2015 at 10:28 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
>
> The only problem is that you don't know how high you need to go in
> general.

Sure, but I didn't know anyway, so no matter what upper bound I choose
(or what function I choose/implement), it's just going to be a guess.
os.closerange just codifies the straightforward procedure.

Skip

[toc] | [prev] | [next] | [standalone]

#91857

Skip Montanaro <skip.montanaro@gmail.com>:

> On Tue, Jun 2, 2015 at 10:28 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
>>
>> The only problem is that you don't know how high you need to go in
>> general.
>
> Sure, but I didn't know anyway, so no matter what upper bound I choose
> (or what function I choose/implement), it's just going to be a guess.
> os.closerange just codifies the straightforward procedure.

Under linux, the cleanest way seems to be going through the files under
/proc/self/fd:

    def close_fds(leave_open=[0, 1, 2]):
        fds = os.listdir(b'/proc/self/fd')
        for fdn in fds:
            fd = int(fdn)
            if fd not in leave_open:
                os.close(fd)

No need for a upper bound.


Marko

[toc] | [prev] | [next] | [standalone]

#91858

On 2015-06-02, Marko Rauhamaa <marko@pacujo.net> wrote:
> Skip Montanaro <skip.montanaro@gmail.com>:
>
>> On Tue, Jun 2, 2015 at 10:28 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
>>>
>>> The only problem is that you don't know how high you need to go in
>>> general.
>>
>> Sure, but I didn't know anyway, so no matter what upper bound I choose
>> (or what function I choose/implement), it's just going to be a guess.
>> os.closerange just codifies the straightforward procedure.
>
> Under linux, the cleanest way seems to be going through the files under
> /proc/self/fd:
>
>     def close_fds(leave_open=[0, 1, 2]):
>         fds = os.listdir(b'/proc/self/fd')
>         for fdn in fds:
>             fd = int(fdn)
>             if fd not in leave_open:
>                 os.close(fd)
>
> No need for a upper bound.

Or use the more generic code that I already posted in this thread:

  def closeall(min=0, max=4096, keep=()):
      """Close all open file descriptors except for the given exceptions.

      Any file descriptors below or equal to `min`, or in the set `keep`
      will not be closed. Any file descriptors above `max` *might* not be
      closed.
      """
      # First try /proc/self/pid
      try:
          for fd in os.listdir("/proc/self/fd"):
              try:
                  fd = int(fd)
              except ValueError:
                  continue
              if fd >= min and fd not in keep:
                  os.close(int(fd))
          return
      except OSError as exc:
          if exc[0] != errno.ENOENT:
              raise
      # If /proc was not available, fall back to closing a lot of descriptors.
      for fd in range(min, max):
          if fd not in keep:
              try:
                  os.close(fd)
              except OSError as exc:
                  if exc[0] != errno.EBADF:
                      raise

This function could use os.closerange(), but if the documentation is
correct and it ignores *all* errors and not just EBADF, then it
sounds like os.closerange() should not in fact ever be used for any
purpose.

[toc] | [prev] | [next] | [standalone]

#92553

The subprocess module uses upper bound MAXFD which is defined as

<code>
try:
    MAXFD = os.sysconf("SC_OPEN_MAX")
except:
    MAXFD = 256
</code>

/Jean

[toc] | [prev] | [next] | [standalone]

#91853

Skip Montanaro <skip.montanaro@gmail.com> writes:

> Reviving (and concluding) a thread I started a couple weeks ago, I asked:
>
>> The basic fork/exec dance is not a problem, but how do I discover
>> all the open file descriptors in the new child process to make sure
>> they get closed? Do I simply start at fd 3 and call os.close() on
>> everything up to some largish fd number?
>
> I wanted this again today (for different reasons than before).
> Googling for "python close all file descriptors" returned the os
> module docs as the first hit, and lo and behold, what do I see
> documented? os.closerange (new in 2.6):
>
>     os.closerange(fd_low, fd_high)
>     Close all file descriptors from fd_low (inclusive) to fd_high
>     (exclusive), ignoring errors.

The close(2) manpage has the following warning on my Linux system:

| Not checking the return value of close() is a common  but  nevertheless
| serious  programming error.  It is quite possible that errors on a pre‐
| vious write(2) operation are first reported at the final close().   Not
| checking the return value when closing the file may lead to silent loss
| of data.  This can especially be observed with NFS and with disk quota.
| 

(I haven't followed the thread, but if your problem is to make sure fds
are closed on exec, you may be better off using the... close-on-exec
flag. Or simply do the bookkeeping.)

-- Alain.

[toc] | [prev] | [next] | [standalone]

#91890

Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid>:

> The close(2) manpage has the following warning on my Linux system:
>
> | Not checking the return value of close() is a common but
> | nevertheless serious programming error. It is quite possible that
> | errors on a previous write(2) operation are first reported at the
> | final close(). Not checking the return value when closing the file
> | may lead to silent loss of data. This can especially be observed
> | with NFS and with disk quota.
> | 
>
> (I haven't followed the thread, but if your problem is to make sure
> fds are closed on exec, you may be better off using the...
> close-on-exec flag. Or simply do the bookkeeping.)

The quoted man page passage is a bit untenable.

First, if close() fails, what's a poor program to do? Try again? How
do you get rid of an obnoxious file descriptor? How would close-on-exec
help? Would exec*() fail?

What if an implicit close() fails on _exit(), will _exit() fail then?
(The man page doesn't allow it.)

The need to close all open file descriptors comes between fork() and
exec*(). The kernel (module) does not see the close() system call unless
the reference count drops to zero. Normally, those function calls
between fork() and exec*() are therefore no-ops.

However, there's no guarantee of that. So the parent process might get
to call close() before the child that is about to call exec*(). Then,
the parent would not get the error that the man page talks about.
Instead, the error goes to the child, which has no reasonable way of
dealing with the situation.

I think having NFS et al postpone their I/O errors till close() is
shifting the blame to the victim.


Marko

[toc] | [prev] | [next] | [standalone]

#91895

Marko Rauhamaa <marko@pacujo.net> writes:

> Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid>:
>
>> The close(2) manpage has the following warning on my Linux system:
>>
>> | Not checking the return value of close() is a common but
>> | nevertheless serious programming error. It is quite possible that
>> | errors on a previous write(2) operation are first reported at the
>> | final close(). Not checking the return value when closing the file
>> | may lead to silent loss of data. This can especially be observed
>> | with NFS and with disk quota.
>> | 
>>
>> (I haven't followed the thread, but if your problem is to make sure
>> fds are closed on exec, you may be better off using the...
>> close-on-exec flag. Or simply do the bookkeeping.)
>
> The quoted man page passage is a bit untenable.
>
> First, if close() fails, what's a poor program to do?

Warn the user? Not assume everything went well?  It all depends on the
application, and what the file descriptor represents.

> Try again?

Could be a good idea on NFS or other kind of mounts.

> How do you get rid of an obnoxious file descriptor?

You don't, you check everything before closing the file, with fsync()
for example.

I've no idea what the OP's program was doing, so I'm not going to split
hairs. I can't imagine why one would like to mass-close an arbitrary set
of file descriptors, and I think APIs like os.closerange() are toxic and
an appeal to sloppy programming.

-- Alain.

[toc] | [prev] | [next] | [standalone]

#91907

Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid>:

> Marko Rauhamaa <marko@pacujo.net> writes:
>> First, if close() fails, what's a poor program to do?
>
> Warn the user? Not assume everything went well? It all depends on the
> application, and what the file descriptor represents.

The problem here is in the system call contract, which is broken.
There's no fix. The man page admonition is just hand-waving without
constructive advice.

>> Try again?
> Could be a good idea on NFS or other kind of mounts.

Maybe close() will fail for ever.

> I can't imagine why one would like to mass-close an arbitrary set of
> file descriptors,

That's standard practice before execking a file. Failure to do that can
seriously hurt the parent process. For example, the parent (or child)
will never read an EOF from file descriptor if its duplicate is open in
an unwitting child process. Also, the number of open files in the system
may grow over all limits or simply waste kernel resources.

Close-on-exec is nice, maybe. However, you don't have control over all
file descriptors. Loggers, high-level library calls and others open
files without the application programmer knowing or having direct
control over.

> and I think APIs like os.closerange() are toxic and an appeal to
> sloppy programming.

And you recommend what instead?


Marko

[toc] | [prev] | [next] | [standalone]

#91925

Marko Rauhamaa <marko@pacujo.net> writes:

> Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid>:
>
>> Marko Rauhamaa <marko@pacujo.net> writes:
>>> First, if close() fails, what's a poor program to do?
>>
>> Warn the user? Not assume everything went well? It all depends on the
>> application, and what the file descriptor represents.
>
> The problem here is in the system call contract, which is broken.
> There's no fix. The man page admonition is just hand-waving without
> constructive advice.
>
>>> Try again?
>> Could be a good idea on NFS or other kind of mounts.
>
> Maybe close() will fail for ever.

Your program has to deal with this, something is going wrong, it can't
just close and go on.

>> I can't imagine why one would like to mass-close an arbitrary set of
>> file descriptors,
>
> That's standard practice before execking a file. [...]

>> and I think APIs like os.closerange() are toxic and an appeal to
>> sloppy programming.
>
> And you recommend what instead?

Keeping them around and closing the ones you own, plus doing whatever is
necessary to have libraires (loggers, database connectors, etc.) finish
properly. And not taking the risk of messing with descriptors your
program is not responsible of.

-- Alain.

[toc] | [prev] | [next] | [standalone]

#91926

Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid>:

> Marko Rauhamaa <marko@pacujo.net> writes:
>> Maybe close() will fail for ever.
>
> Your program has to deal with this, something is going wrong, it can't
> just close and go on.

Here's the deal: the child process is saddled with file descriptors it
never wanted in the first place. It can't decline them. Now you're
saying it can't even dispose of them.

The reason this has been allowed to go on is that everybody just closes
the file descriptors and ignores the possibility or repercussions of a
failure. I haven't read about horror stories of this failing.

I readily admit this is very dirty, but since the API doesn't offer a
clean alternative, there's nothing you can/should do about it.


Marko

[toc] | [prev] | [next] | [standalone]

#91937

On Wed, 03 Jun 2015 10:41:44 +0300, Marko Rauhamaa wrote:

> Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid>:
> 
>> Marko Rauhamaa <marko@pacujo.net> writes:
>>> Maybe close() will fail for ever.
>>
>> Your program has to deal with this, something is going wrong, it can't
>> just close and go on.
> 
> Here's the deal: the child process is saddled with file descriptors it
> never wanted in the first place. It can't decline them. Now you're
> saying it can't even dispose of them.
> 
No You cab dispose of them you just need to warn the user that the action 
did not complete correctly & there may be errors with the data.

Example What does your test editor do if you try to save a file back to a 
USB stick that has been removed? does it simply let you think the file 
has been successfully saved? i hope not.

> The reason this has been allowed to go on is that everybody just closes
> the file descriptors and ignores the possibility or repercussions of a
> failure. I haven't read about horror stories of this failing.
> 
> I readily admit this is very dirty, but since the API doesn't offer a
> clean alternative, there's nothing you can/should do about it.
> 
> 
> Marko





-- 
Never be afraid to tell the world who you are.
		-- Anonymous

[toc] | [prev] | [next] | [standalone]

#91945

On Wed, 3 Jun 2015 07:38 pm, alister wrote:

> On Wed, 03 Jun 2015 10:41:44 +0300, Marko Rauhamaa wrote:
[...]
>> Here's the deal: the child process is saddled with file descriptors it
>> never wanted in the first place. It can't decline them. Now you're
>> saying it can't even dispose of them.
>> 
> No You cab dispose of them you just need to warn the user that the action
> did not complete correctly & there may be errors with the data.

*What* action? Pay closer attention to what Marko wrote:

"the child process ..."

How does the child process know what action didn't complete? What error
message are you going to display to the user?

"Error when closing file descriptor 123456"

What action do you think the user can take on seeing this error message?

 
> Example What does your test editor do if you try to save a file back to a
> USB stick that has been removed? does it simply let you think the file
> has been successfully saved? i hope not.


Correct me if I'm wrong, but don't you get an error when you try to *open* a
file on a missing USB stick?

There are at least three opportunities to get an error when writing a file:

* opening the file;
* writing to the file;
* closing the file.

It's not clear what conditions will lead to the first two succeeding but the
third failing, or what to do if you get such an error. If you don't know
what caused close() to fail, then you can't do anything about it, and if
you can't do anything about it, what's the point in reporting the error to
the user who will just get frustrated and nervous and not be able to do
anything about it either?

I'm sure that there are circumstances where an error when closing a file is
a clear and obvious fault that should be reported. But in the scenario that
Marko is describing, I'm not so sure that is the case.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#91947

On Wed, 03 Jun 2015 22:07:47 +1000, Steven D'Aprano wrote:

> On Wed, 3 Jun 2015 07:38 pm, alister wrote:
> 
>> On Wed, 03 Jun 2015 10:41:44 +0300, Marko Rauhamaa wrote:
> [...]
>>> Here's the deal: the child process is saddled with file descriptors it
>>> never wanted in the first place. It can't decline them. Now you're
>>> saying it can't even dispose of them.
>>> 
>> No You cab dispose of them you just need to warn the user that the
>> action did not complete correctly & there may be errors with the data.
> 
> *What* action? Pay closer attention to what Marko wrote:
> 
> "the child process ..."
> 
> How does the child process know what action didn't complete? What error
> message are you going to display to the user?
> 
> "Error when closing file descriptor 123456"
> 
> What action do you think the user can take on seeing this error message?
> 
> 
>> Example What does your test editor do if you try to save a file back to
>> a USB stick that has been removed? does it simply let you think the
>> file has been successfully saved? i hope not.
> 
> 
> Correct me if I'm wrong, but don't you get an error when you try to
> *open* a file on a missing USB stick?
> 
> There are at least three opportunities to get an error when writing a
> file:
> 
> * opening the file;
> * writing to the file;
> * closing the file.
> 
> It's not clear what conditions will lead to the first two succeeding but
> the third failing, or what to do if you get such an error. If you don't
> know what caused close() to fail,

open file for appending data
write some data
remove mem stick
now try to close file

> then you can't do anything about it,
Correct you cant do anything about it.
> and if you can't do anything about it, what's the point in reporting the
> error to the user who will just get frustrated and nervous and not be
> able to do anything about it either?

Because they need to know they may have just lost important data whilst 
they still have a chance to do it again. finding out 6 months later that 
they had screwed up saving data on a multi-million pound account will not 
be good & quite rightly considered to be due to a bug in your software.


> 
> I'm sure that there are circumstances where an error when closing a file
> is a clear and obvious fault that should be reported. But in the
> scenario that Marko is describing, I'm not so sure that is the case.

from the scenario Marco is reporting I get the impression that he is 
having to interact with a system that is fundamentally flawed from the 
ground up.



-- 
Standards are crucial.  And the best thing about standards is: there are
so ____many to choose from!

[toc] | [prev] | [next] | [standalone]

#91952

alister <alister.nospam.ware@ntlworld.com>:

> from the scenario Marco is reporting I get the impression that he is
> having to interact with a system that is fundamentally flawed from the
> ground up.

Well, yes. It's called linux, but it's not all bad. I just think that
man page was being sanctimonious.


Marko

[toc] | [prev] | [next] | [standalone]

#91955

On Wed, 03 Jun 2015 15:27:19 +0300, Marko Rauhamaa wrote:

> alister <alister.nospam.ware@ntlworld.com>:
> 
>> from the scenario Marco is reporting I get the impression that he is
>> having to interact with a system that is fundamentally flawed from the
>> ground up.
> 
> Well, yes. It's called linux, but it's not all bad. I just think that
> man page was being sanctimonious.
> 
> 
> Marko

I meant the program that is supplying your app with file handles willy- 
nilly without caring what happens to them



-- 
"...a most excellent barbarian ... Genghis Kahn!"
-- _Bill And Ted's Excellent Adventure_

[toc] | [prev] | [next] | [standalone]

#91970

alister <alister.nospam.ware@ntlworld.com>:

> I meant the program that is supplying your app with file handles
> willy- nilly without caring what happens to them

You seem to be advocating a strategy whereby the application keeps close
track of all file descriptors and closes them individually as needed.

Thing is, processes can be forked by library calls. For example, you
might have developed a class that converts a synchronous database API
into an asynchronous one. You'll have your object fork a helper process
that makes blocking calls while the object methods make sure never to
block the caller.

The application doesn't know the library would be starting a child
process. On the other hand, the library has no idea on what files the
application might have open. That's why the library traverses all file
descriptors and closes them categorically after forking.


Marko

[toc] | [prev] | [next] | [standalone]

#91951

Steven D'Aprano <steve@pearwood.info>:

> How does the child process know what action didn't complete? What
> error message are you going to display to the user?
>
> "Error when closing file descriptor 123456"
>
> What action do you think the user can take on seeing this error
> message?

Besides, even if you wanted to display that error message, you might not
be able to since maybe you have unknowingly closed fd 123433, which is
the hidden connection to the display server. So the program might just
crash in the attempt. Or the hidden fd is still open but won't tolerate
two processes sharing the connection.

You definitely must not use sys.stderr from the child process nor are
you allowed to exit through an exception but must call os._exit()
instead.

The child process is highly limited in what it can do between os.fork()
and os.execve(). To put it simply, it must not call any library
functions except system calls.

You might be able to reserve a special pipe to convey such errors to the
parent. Only don't rely on getting an EOF as closing the pipe might
fail...


Marko

[toc] | [prev] | [next] | [standalone]

#91954

On Wed, Jun 3, 2015, at 08:25, Marko Rauhamaa wrote:
> Steven D'Aprano <steve@pearwood.info>:
> 
> > How does the child process know what action didn't complete? What
> > error message are you going to display to the user?
> 
> You definitely must not use sys.stderr from the child process nor are
> you allowed to exit through an exception but must call os._exit()
> instead.

Why does the child process need to report the error at all? The parent
process will find out naturally when *it* tries to close the same file
descriptor.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python

csiph-web