Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!news-transit.tcx.org.uk!nntp-feed.chiark.greenend.org.uk!ewrotcd!news.nosignal.org!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path: <python-python-list@m.gmane.org>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.005
X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'bsd': 0.05; 'flags': 0.05; 'python': 0.08; 'descriptors': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:80.91.229.12': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'received:lo.gmane.org': 0.09; 'subject:files': 0.09; 'files.': 0.09; '1024': 0.16; 'did:': 0.16; 'freshly': 0.16; 'limit.': 0.16; 'tobias': 0.16; 'linux': 0.17; 'appropriate': 0.19; 'trying': 0.21; 'options.': 0.21; "doesn't": 0.23; 'header:In-Reply-To:1': 0.23; "i'm": 0.26; 'subject:number': 0.30; 'installation': 0.30; 'familiar': 0.32; 'usually': 0.32; 'to:addr:python-list': 0.32; "i've": 0.33; 'header:User-Agent:1': 0.33; 'header:X-Complaints- To:1': 0.33; 'convince': 0.34; 'skip:k 20': 0.34; 'similar': 0.35; 'subject:/': 0.35; 'file': 0.36; 'but': 0.37; 'options': 0.37; 'received:org': 0.37; 'open': 0.38; "it's": 0.39; 'why': 0.39; 'subject:: ': 0.39; 'might': 0.39; 'to:addr:python.org': 0.39; 'happens': 0.40; 'data': 0.40; 'files': 0.40; 'more': 0.60; 'total': 0.61; 'our': 0.63; 'here': 0.65; 'high': 0.66; 'limit': 0.67; 'unlimited': 0.68; '100': 0.69; 'biggest': 0.74; 'million': 0.76; '200k': 0.84; 'schrieb': 0.84
X-Injected-Via-Gmane: http://gmane.org/
To: python-list@python.org
From: Christian Heimes <lists@cheimes.de>
Subject: Re: Py2.7/FreeBSD: maximum number of open files
Date: Mon, 14 Nov 2011 17:25:13 +0100
References: <634914A010D0B943A035D226786325D42D0C264772@EXVMBX020-12.exch020.serverdata.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Gmane-NNTP-Posting-Host: smtp.semantics.de
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20110929 Thunderbird/7.0.1
In-Reply-To: <634914A010D0B943A035D226786325D42D0C264772@EXVMBX020-12.exch020.serverdata.net>
X-Enigmail-Version: 1.4a1pre
OpenPGP: id=AD16AB1B; url=http://cheimes.de/heimes.asc
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.2699.1321287931.27778.python-list@python.org>
Lines: 30
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1321287931 news.xs4all.nl 6967 [2001:888:2000:d::a6]:44227
X-Complaints-To: abuse@xs4all.nl
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:15673

Am 14.11.2011 16:57, schrieb Tobias Oberstein:
> I am trying to convince Python to open more than 32k files .. this is on FreeBSD.
> 
> Now I know I have to set appropriate limits .. I did:
> 
> $ sysctl kern.maxfiles
> kern.maxfiles: 204800
> $ sysctl kern.maxfilesperproc
> kern.maxfilesperproc: 200000
> $ sysctl kern.maxvnodes
> kern.maxvnodes: 200000
> $ ulimit
> unlimited
> 
> Here is what happens with a Python freshly built from sources .. it'll tell me I can open 200k files .. but will bail out at 32k:

I'm not familiar with BSD but Linux has similar Kernel options. The
kernel options might be *global* flags to set the total upper limit of
open file descriptors for the entire system, not for a single process.
Also on Linux "ulimit" doesn't display the fd limit. You have to use
"ulimit -n".

Why do you need more than 32k file descriptors anyway? It's an insanely
high amount of FDs. Most programs need less than 100 and the default
value of 1024 on my Linux servers is usually high enough. I've never
increased the fd limit over 8192 and our biggest installation servers
more than 80 TB data in about 20 to 25 million files.

Christian