AW: Py2.7/FreeBSD: maximum number of open files

Path	csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!news.albasani.net!feeder.erje.net!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path	<tobias.oberstein@tavendo.de>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.000
X-Spam-Evidence	'H': 1.00; 'S': 0.00; 'bug': 0.02; '2.7': 0.04; 'happily': 0.07; 'python': 0.08; 'seriously.': 0.09; 'sockets': 0.09; 'subject:files': 0.09; 'anyway': 0.09; 'api': 0.09; 'files.': 0.09; 'case.': 0.15; 'to:name:python-list@python.org': 0.15; 'backported': 0.16; 'int.': 0.16; 'libc': 0.16; 'open()': 0.16; 'overflow': 0.16; 'overflow.': 0.16; 'posix': 0.16; 'strange:': 0.16; 'switching': 0.16; 'twisted': 0.16; 'seems': 0.19; 'to:2**1': 0.19; 'file,': 0.21; 'stuff': 0.22; 'header:In- Reply-To:1': 0.23; 'there.': 0.24; 'code': 0.25; '(in': 0.26; "i'm": 0.26; 'compile': 0.28; 'fixed': 0.28; 'subject:number': 0.30; 'thanks': 0.31; 'signed': 0.31; 'yet': 0.32; 'to:addr :python-list': 0.32; "i've": 0.33; 'there': 0.33; 'it?': 0.33; 'received:169': 0.34; 'received:169.254': 0.34; 'thus,': 0.34; 'however,': 0.34; 'running': 0.35; 'subject:/': 0.35; 'file': 0.36; '...': 0.36; 'charset:us-ascii': 0.36; 'url:pipermail': 0.36; 'but': 0.37; 'using': 0.37; 'configured': 0.38; 'open': 0.38; 'some': 0.38; 'perhaps': 0.38; 'url:org': 0.38; 'i.e.': 0.38; 'subject:: ': 0.39; 'hit': 0.39; 'might': 0.39; 'to:addr:python.org': 0.39; 'files': 0.40; 'more': 0.60; 'your': 0.61; 'limit': 0.67; '100': 0.69; 'soon': 0.72; 'service)': 0.84
From	Tobias Oberstein <tobias.oberstein@tavendo.de>
To	Jon Clements <joncle@googlemail.com>, "python-list@python.org" <python-list@python.org>
Date	Mon, 14 Nov 2011 09:46:00 -0800
Subject	AW: Py2.7/FreeBSD: maximum number of open files
Thread-Topic	Py2.7/FreeBSD: maximum number of open files
Thread-Index	Acyi9CSGl3GUuNdGQc+OSHHHXpW27AAAByyg
References	<634914A010D0B943A035D226786325D42D0C264772@EXVMBX020-12.exch020.serverdata.net> <j9rfda$mfs$1@dough.gmane.org> <634914A010D0B943A035D226786325D42D0C2647C8@EXVMBX020-12.exch020.serverdata.net> <4EC1470B.90403@cheimes.de> <mailman.2705.1321290230.27778.python-list@python.org> <e3901b26-33ab-41b4-a8b1-82d08f15c125@u6g2000vbg.googlegroups.com>
In-Reply-To	<e3901b26-33ab-41b4-a8b1-82d08f15c125@u6g2000vbg.googlegroups.com>
Accept-Language	de-DE, en-US
Content-Language	de-DE
X-MS-Has-Attach
X-MS-TNEF-Correlator
acceptlanguage	de-DE, en-US
Content-Type	text/plain; charset="us-ascii"
Content-Transfer-Encoding	quoted-printable
MIME-Version	1.0
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.12
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.2707.1321292764.27778.python-list@python.org> (permalink)
Lines	50
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1321292765 news.xs4all.nl 6911 [2001:888:2000:d::a6]:40571
X-Complaints-To	abuse@xs4all.nl
Xref	x330-a1.tempe.blueboxinc.net comp.lang.python:15681

Show key headers only | View raw

> > > > I need 50k sockets + 100 files.
> >
> > > > Thus, this is even more strange: the Python (a Twisted service)
> > > > will happily accept 50k sockets, but as soon as you do open() a file, it'll
> bail out.
> >
> > > A limit of 32k smells like a overflow in a signed int. Perhaps your
> > > system is able and configured to handle more than 32k FDs but you
> > > hit an artificial limit because some C code or API has a overflow.
> > > This seems to be a known bug in
> > > FreeBSDhttp://lists.freebsd.org/pipermail/freebsd-bugs/2010-
> > > July/040689.html
> >
> > This is unbelievable.
> >
> > I've just tested: the bug (in libc) is still there on FreeBSD 8.2 p3
> > ... both on i386 _and_ amd64.
> >
> > Now I'm f***d;(
> >
> > A last chance: is it possible to compile Python for not using libc
> > fopen(), but the Posix open()?
> >
> > Thanks anyway for this hint!
> 
> Have you tried/or is it possible to get your 100 or whatever files first, before
> your sockets?

If I only needed to open a fixed set of files, that might be a workaround.

However, this is not the case.

I.e. Twisted will do log switching and create/open a new file when the 50k sockets
are already there.

I just confirmed that the bug is even there for FreeBSD 9 RC1 !

This is most unfortunate. Seriously.

I am running out of options, since I am willing to make my stuff Python 3 compatible,
but Twisted is not yet there.

Using the backported new IO on Python 2.7 will not make open() automatically use the new IO, will it?

Thread

AW: Py2.7/FreeBSD: maximum number of open files Tobias Oberstein <tobias.oberstein@tavendo.de> - 2011-11-14 09:03 -0800
  Re: Py2.7/FreeBSD: maximum number of open files Jon Clements <joncle@googlemail.com> - 2011-11-14 09:33 -0800
    AW: Py2.7/FreeBSD: maximum number of open files Tobias Oberstein <tobias.oberstein@tavendo.de> - 2011-11-14 09:46 -0800
    Re: Py2.7/FreeBSD: maximum number of open files Christian Heimes <lists@cheimes.de> - 2011-11-14 19:03 +0100
    AW: Py2.7/FreeBSD: maximum number of open files Tobias Oberstein <tobias.oberstein@tavendo.de> - 2011-11-14 10:28 -0800
    Re: Py2.7/FreeBSD: maximum number of open files Christian Heimes <lists@cheimes.de> - 2011-11-14 20:34 +0100
      Re: Py2.7/FreeBSD: maximum number of open files Chris Torek <nospam@torek.net> - 2011-11-30 19:29 +0000

csiph-web