Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #15681

AW: Py2.7/FreeBSD: maximum number of open files

Show all headers | View raw

> > > > I need 50k sockets + 100 files.
> >
> > > > Thus, this is even more strange: the Python (a Twisted service)
> > > > will happily accept 50k sockets, but as soon as you do open() a file, it'll
> bail out.
> >
> > > A limit of 32k smells like a overflow in a signed int. Perhaps your
> > > system is able and configured to handle more than 32k FDs but you
> > > hit an artificial limit because some C code or API has a overflow.
> > > This seems to be a known bug in
> > > FreeBSDhttp://lists.freebsd.org/pipermail/freebsd-bugs/2010-
> > > July/040689.html
> >
> > This is unbelievable.
> >
> > I've just tested: the bug (in libc) is still there on FreeBSD 8.2 p3
> > ... both on i386 _and_ amd64.
> >
> > Now I'm f***d;(
> >
> > A last chance: is it possible to compile Python for not using libc
> > fopen(), but the Posix open()?
> >
> > Thanks anyway for this hint!
> 
> Have you tried/or is it possible to get your 100 or whatever files first, before
> your sockets?

If I only needed to open a fixed set of files, that might be a workaround.

However, this is not the case.

I.e. Twisted will do log switching and create/open a new file when the 50k sockets
are already there.

I just confirmed that the bug is even there for FreeBSD 9 RC1 !

This is most unfortunate. Seriously.

I am running out of options, since I am willing to make my stuff Python 3 compatible,
but Twisted is not yet there.

Using the backported new IO on Python 2.7 will not make open() automatically use the new IO, will it?

 

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

AW: Py2.7/FreeBSD: maximum number of open files Tobias Oberstein <tobias.oberstein@tavendo.de> - 2011-11-14 09:03 -0800
  Re: Py2.7/FreeBSD: maximum number of open files Jon Clements <joncle@googlemail.com> - 2011-11-14 09:33 -0800
    AW: Py2.7/FreeBSD: maximum number of open files Tobias Oberstein <tobias.oberstein@tavendo.de> - 2011-11-14 09:46 -0800
    Re: Py2.7/FreeBSD: maximum number of open files Christian Heimes <lists@cheimes.de> - 2011-11-14 19:03 +0100
    AW: Py2.7/FreeBSD: maximum number of open files Tobias Oberstein <tobias.oberstein@tavendo.de> - 2011-11-14 10:28 -0800
    Re: Py2.7/FreeBSD: maximum number of open files Christian Heimes <lists@cheimes.de> - 2011-11-14 20:34 +0100
      Re: Py2.7/FreeBSD: maximum number of open files Chris Torek <nospam@torek.net> - 2011-11-30 19:29 +0000

csiph-web