Re: Py2.7/FreeBSD: maximum number of open files

Path	csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path	<lists@cheimes.de>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.004
X-Spam-Evidence	'H': 0.99; 'S': 0.00; 'bug': 0.02; 'happily': 0.07; 'python': 0.08; 'files:': 0.09; 'sockets': 0.09; 'subject:files': 0.09; 'api': 0.09; 'files.': 0.09; 'tries': 0.15; 'cc:addr:python-list': 0.15; 'int.': 0.16; 'open()': 0.16; 'overflow': 0.16; 'overflow.': 0.16; 'strange:': 0.16; 'tobias': 0.16; 'twisted': 0.16; 'contribute': 0.18; 'seems': 0.19; 'connections': 0.21; 'file,': 0.21; 'header:In-Reply-To:1': 0.23; 'cc:2**0': 0.25; 'code': 0.25; "i'm": 0.26; 'beyond': 0.27; 'cc:addr:python.org': 0.29; 'subject:number': 0.30; 'signed': 0.31; 'point,': 0.32; 'header:User-Agent:1': 0.33; 'thus,': 0.34; 'subject:/': 0.35; 'url:pipermail': 0.36; 'doing': 0.37; 'machine': 0.37; 'but': 0.37; 'received:org': 0.37; 'configured': 0.38; 'open': 0.38; 'some': 0.38; 'else': 0.38; 'perhaps': 0.38; 'url:org': 0.38; "it's": 0.39; 'subject:: ': 0.39; 'hit': 0.39; 'did': 0.40; 'more': 0.60; 'your': 0.61; 'dedicated': 0.63; 'here.': 0.66; 'limit': 0.67; '100': 0.69; 'soon': 0.72; 'schrieb': 0.84; 'service)': 0.84
Date	Mon, 14 Nov 2011 17:51:23 +0100
From	Christian Heimes <lists@cheimes.de>
User-Agent	Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20110929 Thunderbird/7.0.1
MIME-Version	1.0
To	Tobias Oberstein <tobias.oberstein@tavendo.de>
Subject	Re: Py2.7/FreeBSD: maximum number of open files
References	<634914A010D0B943A035D226786325D42D0C264772@EXVMBX020-12.exch020.serverdata.net> <j9rfda$mfs$1@dough.gmane.org> <634914A010D0B943A035D226786325D42D0C2647C8@EXVMBX020-12.exch020.serverdata.net>
In-Reply-To	<634914A010D0B943A035D226786325D42D0C2647C8@EXVMBX020-12.exch020.serverdata.net>
X-Enigmail-Version	1.4a1pre
Content-Type	text/plain; charset=ISO-8859-1
Content-Transfer-Encoding	7bit
Cc	"python-list@python.org" <python-list@python.org>
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.12
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.2704.1321289996.27778.python-list@python.org> (permalink)
Lines	33
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1321289996 news.xs4all.nl 6917 [2001:888:2000:d::a6]:38420
X-Complaints-To	abuse@xs4all.nl
Xref	x330-a1.tempe.blueboxinc.net comp.lang.python:15676

Show key headers only | View raw

Am 14.11.2011 17:36, schrieb Tobias Oberstein:
> This is a dedicated machine doing nothing else .. I'm monitoring global FD usage
> 
> sysctl kern.openfiles
> 
> and it's way beyond the configured limit
> 
> $ ulimit -n
> 200000

Apparently you did everything right here. Well, it was worth the try. ;)


> It's not for files:
> 
> This is a network service .. I tested it with up to 50k TCP connections .. however
> at this point, when the service tries to open a file, it'll bail out.
> 
> Sockets+Files both contribute to open FDs.
> 
> I need 50k sockets + 100 files.
> 
> Thus, this is even more strange: the Python (a Twisted service) will happily
> accept 50k sockets, but as soon as you do open() a file, it'll bail out.

A limit of 32k smells like a overflow in a signed int. Perhaps your
system is able and configured to handle more than 32k FDs but you hit an
artificial limit because some C code or API has a overflow. This seems
to be a known bug in FreeBSD
http://lists.freebsd.org/pipermail/freebsd-bugs/2010-July/040689.html


Christian

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: Py2.7/FreeBSD: maximum number of open files Christian Heimes <lists@cheimes.de> - 2011-11-14 17:51 +0100

csiph-web