Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #15677 > unrolled thread

AW: Py2.7/FreeBSD: maximum number of open files

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  AW: Py2.7/FreeBSD: maximum number of open files Tobias Oberstein <tobias.oberstein@tavendo.de> - 2011-11-14 09:03 -0800
    Re: Py2.7/FreeBSD: maximum number of open files Jon Clements <joncle@googlemail.com> - 2011-11-14 09:33 -0800
      AW: Py2.7/FreeBSD: maximum number of open files Tobias Oberstein <tobias.oberstein@tavendo.de> - 2011-11-14 09:46 -0800
      Re: Py2.7/FreeBSD: maximum number of open files Christian Heimes <lists@cheimes.de> - 2011-11-14 19:03 +0100
      AW: Py2.7/FreeBSD: maximum number of open files Tobias Oberstein <tobias.oberstein@tavendo.de> - 2011-11-14 10:28 -0800
      Re: Py2.7/FreeBSD: maximum number of open files Christian Heimes <lists@cheimes.de> - 2011-11-14 20:34 +0100
        Re: Py2.7/FreeBSD: maximum number of open files Chris Torek <nospam@torek.net> - 2011-11-30 19:29 +0000

#15677 — AW: Py2.7/FreeBSD: maximum number of open files

> > I need 50k sockets + 100 files.
> >
> > Thus, this is even more strange: the Python (a Twisted service) will
> > happily accept 50k sockets, but as soon as you do open() a file, it'll bail out.
> 
> A limit of 32k smells like a overflow in a signed int. Perhaps your system is
> able and configured to handle more than 32k FDs but you hit an artificial limit
> because some C code or API has a overflow. This seems to be a known bug in
> FreeBSD http://lists.freebsd.org/pipermail/freebsd-bugs/2010-
> July/040689.html

This is unbelievable.

I've just tested: the bug (in libc) is still there on FreeBSD 8.2 p3 ... both on i386
_and_ amd64.

Now I'm f***d;(

A last chance: is it possible to compile Python for not using libc fopen(),
but the Posix open()?

Thanks anyway for this hint!

[toc] | [next] | [standalone]

#15680 — Re: Py2.7/FreeBSD: maximum number of open files

On Nov 14, 5:03 pm, Tobias Oberstein <tobias.oberst...@tavendo.de>
wrote:
> > > I need 50k sockets + 100 files.
>
> > > Thus, this is even more strange: the Python (a Twisted service) will
> > > happily accept 50k sockets, but as soon as you do open() a file, it'll bail out.
>
> > A limit of 32k smells like a overflow in a signed int. Perhaps your system is
> > able and configured to handle more than 32k FDs but you hit an artificial limit
> > because some C code or API has a overflow. This seems to be a known bug in
> > FreeBSDhttp://lists.freebsd.org/pipermail/freebsd-bugs/2010-
> > July/040689.html
>
> This is unbelievable.
>
> I've just tested: the bug (in libc) is still there on FreeBSD 8.2 p3 ... both on i386
> _and_ amd64.
>
> Now I'm f***d;(
>
> A last chance: is it possible to compile Python for not using libc fopen(),
> but the Posix open()?
>
> Thanks anyway for this hint!

Have you tried/or is it possible to get your 100 or whatever files
first, before your sockets?

hth

Jon

[toc] | [prev] | [next] | [standalone]

#15681

> > > > I need 50k sockets + 100 files.
> >
> > > > Thus, this is even more strange: the Python (a Twisted service)
> > > > will happily accept 50k sockets, but as soon as you do open() a file, it'll
> bail out.
> >
> > > A limit of 32k smells like a overflow in a signed int. Perhaps your
> > > system is able and configured to handle more than 32k FDs but you
> > > hit an artificial limit because some C code or API has a overflow.
> > > This seems to be a known bug in
> > > FreeBSDhttp://lists.freebsd.org/pipermail/freebsd-bugs/2010-
> > > July/040689.html
> >
> > This is unbelievable.
> >
> > I've just tested: the bug (in libc) is still there on FreeBSD 8.2 p3
> > ... both on i386 _and_ amd64.
> >
> > Now I'm f***d;(
> >
> > A last chance: is it possible to compile Python for not using libc
> > fopen(), but the Posix open()?
> >
> > Thanks anyway for this hint!
> 
> Have you tried/or is it possible to get your 100 or whatever files first, before
> your sockets?

If I only needed to open a fixed set of files, that might be a workaround.

However, this is not the case.

I.e. Twisted will do log switching and create/open a new file when the 50k sockets
are already there.

I just confirmed that the bug is even there for FreeBSD 9 RC1 !

This is most unfortunate. Seriously.

I am running out of options, since I am willing to make my stuff Python 3 compatible,
but Twisted is not yet there.

Using the backported new IO on Python 2.7 will not make open() automatically use the new IO, will it?

 

[toc] | [prev] | [next] | [standalone]

#15684 — Re: Py2.7/FreeBSD: maximum number of open files

Am 14.11.2011 18:46, schrieb Tobias Oberstein:
> I just confirmed that the bug is even there for FreeBSD 9 RC1 !
> 
> This is most unfortunate. Seriously.

W00t, that sucks! You could migrate to another BSD (NetBSD) or Linux ... :)

> I am running out of options, since I am willing to make my stuff Python 3 compatible,
> but Twisted is not yet there.
> 
> Using the backported new IO on Python 2.7 will not make open() automatically use the new IO, will it?

No, the open() function of Python 2.7 will still use the file class
which in return uses fopen(). You could try to monkey patch the built-in
open() function. It's mostly API compatible with the current open()
function:

  >>> import io, __builtin__
  >>> __builtin__.open = io.open

It works as long as no codes checks for isinstance(obj, file). If your
app only has to worry about log files, you might want to overwrite the
_open() method of logging.FileHandler and its subclasses.

Christian

[toc] | [prev] | [next] | [standalone]

#15686

> > I just confirmed that the bug is even there for FreeBSD 9 RC1 !
> >
> > This is most unfortunate. Seriously.
> 
> W00t, that sucks! You could migrate to another BSD (NetBSD) or Linux ... :)

No, thanks;)

> > I am running out of options, since I am willing to make my stuff
> > Python 3 compatible, but Twisted is not yet there.
> >
> > Using the backported new IO on Python 2.7 will not make open()
> automatically use the new IO, will it?
> 
> No, the open() function of Python 2.7 will still use the file class which in
> return uses fopen(). You could try to monkey patch the built-in
> open() function. It's mostly API compatible with the current open()
> function:
> 
>   >>> import io, __builtin__
>   >>> __builtin__.open = io.open
> 
> It works as long as no codes checks for isinstance(obj, file). If your app only
> has to worry about log files, you might want to overwrite the
> _open() method of logging.FileHandler and its subclasses.
> 

Thanks! This is probably the most practical option I can go.

I've just tested: the backported new IO on Python 2.7 will indeed
open >32k files on FreeBSD. It also creates the files much faster.
The old, non-monkey-patched version was getting slower and
slower as more files were opened/created ..

There seem to be slight differences though:

Non-monkey patched: I can write to the file a non-Unicode string,
even when the file was opened non-Binary.

With monkey patch: either open the file Binary-mode, or
write Unicode strings ..

I need to see if / what breaks in Twisted.

I can handle my own code .. no problem.

Thanks alot!!



import io, __builtin__
__builtin__.open = io.open

import resource

max = resource.getrlimit(resource.RLIMIT_NOFILE)
cnt = 0
print "maximum FDs", max

max = 33000

fds = []

while cnt < max:
   f = open("/tmp/test1/test_%d" % cnt, "wb")
   f.write("test")
   fds.append(f)
   cnt += 1
   if cnt % 1000 == 0:
      print "opened %d files" % cnt

print "ok, created %d files" % cnt

[toc] | [prev] | [next] | [standalone]

#15688 — Re: Py2.7/FreeBSD: maximum number of open files

Am 14.11.2011 19:28, schrieb Tobias Oberstein:
> Thanks! This is probably the most practical option I can go.
> 
> I've just tested: the backported new IO on Python 2.7 will indeed
> open >32k files on FreeBSD. It also creates the files much faster.
> The old, non-monkey-patched version was getting slower and
> slower as more files were opened/created ..

I wonder what's causing the O(n^2) behavior. Is it the old file type or
BSD's fopen() fault?


> There seem to be slight differences though:
> 
> Non-monkey patched: I can write to the file a non-Unicode string,
> even when the file was opened non-Binary.
> 
> With monkey patch: either open the file Binary-mode, or
> write Unicode strings ..

Python 3.x doesn't collate data and text but distinguishes between bytes
(str type in Python 2.x) and unicode. You can't write unicode to a
binary file and str to a text file. See it as an opportunity! You are
taking the first step towards Python 3 compatibility. :)

Christian

[toc] | [prev] | [next] | [standalone]

#16448 — Re: Py2.7/FreeBSD: maximum number of open files

In article <mailman.2711.1321299276.27778.python-list@python.org>
Christian Heimes  <lists@cheimes.de> wrote:
>Am 14.11.2011 19:28, schrieb Tobias Oberstein:
>> Thanks! This is probably the most practical option I can go.
>> 
>> I've just tested: the backported new IO on Python 2.7 will indeed
>> open >32k files on FreeBSD. It also creates the files much faster.
>> The old, non-monkey-patched version was getting slower and
>> slower as more files were opened/created ..
>
>I wonder what's causing the O(n^2) behavior. Is it the old file type or
>BSD's fopen() fault?

It is code in libc.  My old stdio (still in use on FreeBSD) was
never designed to be used in situations with more than roughly 256
file descriptors -- hence the "short" in the file number field.
(The OS used to be full of other places that kept the maximum
allowable file descriptor fairly small, such as the on-stack copies
of fd_set objects in select() calls.)

You will want to redesign the code that finds or creates a free
"FILE" object, and probably some of the things that work with
line-buffered FILEs (specifically calls to _fwalk() when reading
from a line-buffered stream).
-- 
In-Real-Life: Chris Torek, Wind River Systems
Intel require I note that my opinions are not those of WRS or Intel
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W)  +1 801 277 2603
email: gmail (figure it out)      http://web.torek.net/torek/index.html

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web