Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!news.albasani.net!feeder.erje.net!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path: <tobias.oberstein@tavendo.de>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.000
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'bug': 0.02; '2.7': 0.04; 'happily': 0.07; 'python': 0.08; 'seriously.': 0.09; 'sockets': 0.09; 'subject:files': 0.09; 'anyway': 0.09; 'api': 0.09; 'files.': 0.09; 'case.': 0.15; 'to:name:python-list@python.org': 0.15; 'backported': 0.16; 'int.': 0.16; 'libc': 0.16; 'open()': 0.16; 'overflow': 0.16; 'overflow.': 0.16; 'posix': 0.16; 'strange:': 0.16; 'switching': 0.16; 'twisted': 0.16; 'seems': 0.19; 'to:2**1': 0.19; 'file,': 0.21; 'stuff': 0.22; 'header:In- Reply-To:1': 0.23; 'there.': 0.24; 'code': 0.25; '(in': 0.26; "i'm": 0.26; 'compile': 0.28; 'fixed': 0.28; 'subject:number': 0.30; 'thanks': 0.31; 'signed': 0.31; 'yet': 0.32; 'to:addr :python-list': 0.32; "i've": 0.33; 'there': 0.33; 'it?': 0.33; 'received:169': 0.34; 'received:169.254': 0.34; 'thus,': 0.34; 'however,': 0.34; 'running': 0.35; 'subject:/': 0.35; 'file': 0.36; '...': 0.36; 'charset:us-ascii': 0.36; 'url:pipermail': 0.36; 'but': 0.37; 'using': 0.37; 'configured': 0.38; 'open': 0.38; 'some': 0.38; 'perhaps': 0.38; 'url:org': 0.38; 'i.e.': 0.38; 'subject:: ': 0.39; 'hit': 0.39; 'might': 0.39; 'to:addr:python.org': 0.39; 'files': 0.40; 'more': 0.60; 'your': 0.61; 'limit': 0.67; '100': 0.69; 'soon': 0.72; 'service)': 0.84
From: Tobias Oberstein <tobias.oberstein@tavendo.de>
To: Jon Clements <joncle@googlemail.com>, "python-list@python.org" <python-list@python.org>
Date: Mon, 14 Nov 2011 09:46:00 -0800
Subject: AW: Py2.7/FreeBSD: maximum number of open files
Thread-Topic: Py2.7/FreeBSD: maximum number of open files
Thread-Index: Acyi9CSGl3GUuNdGQc+OSHHHXpW27AAAByyg
References: <634914A010D0B943A035D226786325D42D0C264772@EXVMBX020-12.exch020.serverdata.net> <j9rfda$mfs$1@dough.gmane.org> <634914A010D0B943A035D226786325D42D0C2647C8@EXVMBX020-12.exch020.serverdata.net> <4EC1470B.90403@cheimes.de> <mailman.2705.1321290230.27778.python-list@python.org> <e3901b26-33ab-41b4-a8b1-82d08f15c125@u6g2000vbg.googlegroups.com>
In-Reply-To: <e3901b26-33ab-41b4-a8b1-82d08f15c125@u6g2000vbg.googlegroups.com>
Accept-Language: de-DE, en-US
Content-Language: de-DE
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: de-DE, en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.2707.1321292764.27778.python-list@python.org>
Lines: 50
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1321292765 news.xs4all.nl 6911 [2001:888:2000:d::a6]:40571
X-Complaints-To: abuse@xs4all.nl
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:15681

> > > > I need 50k sockets + 100 files.
> >
> > > > Thus, this is even more strange: the Python (a Twisted service)
> > > > will happily accept 50k sockets, but as soon as you do open() a fil=
e, it'll
> bail out.
> >
> > > A limit of 32k smells like a overflow in a signed int. Perhaps your
> > > system is able and configured to handle more than 32k FDs but you
> > > hit an artificial limit because some C code or API has a overflow.
> > > This seems to be a known bug in
> > > FreeBSDhttp://lists.freebsd.org/pipermail/freebsd-bugs/2010-
> > > July/040689.html
> >
> > This is unbelievable.
> >
> > I've just tested: the bug (in libc) is still there on FreeBSD 8.2 p3
> > ... both on i386 _and_ amd64.
> >
> > Now I'm f***d;(
> >
> > A last chance: is it possible to compile Python for not using libc
> > fopen(), but the Posix open()?
> >
> > Thanks anyway for this hint!
>=20
> Have you tried/or is it possible to get your 100 or whatever files first,=
 before
> your sockets?

If I only needed to open a fixed set of files, that might be a workaround.

However, this is not the case.

I.e. Twisted will do log switching and create/open a new file when the 50k =
sockets
are already there.

I just confirmed that the bug is even there for FreeBSD 9 RC1 !

This is most unfortunate. Seriously.

I am running out of options, since I am willing to make my stuff Python 3 c=
ompatible,
but Twisted is not yet there.

Using the backported new IO on Python 2.7 will not make open() automaticall=
y use the new IO, will it?

=20