Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #16448

Re: Py2.7/FreeBSD: maximum number of open files

Show all headers | View raw

In article <mailman.2711.1321299276.27778.python-list@python.org>
Christian Heimes  <lists@cheimes.de> wrote:
>Am 14.11.2011 19:28, schrieb Tobias Oberstein:
>> Thanks! This is probably the most practical option I can go.
>> 
>> I've just tested: the backported new IO on Python 2.7 will indeed
>> open >32k files on FreeBSD. It also creates the files much faster.
>> The old, non-monkey-patched version was getting slower and
>> slower as more files were opened/created ..
>
>I wonder what's causing the O(n^2) behavior. Is it the old file type or
>BSD's fopen() fault?

It is code in libc.  My old stdio (still in use on FreeBSD) was
never designed to be used in situations with more than roughly 256
file descriptors -- hence the "short" in the file number field.
(The OS used to be full of other places that kept the maximum
allowable file descriptor fairly small, such as the on-stack copies
of fd_set objects in select() calls.)

You will want to redesign the code that finds or creates a free
"FILE" object, and probably some of the things that work with
line-buffered FILEs (specifically calls to _fwalk() when reading
from a line-buffered stream).
-- 
In-Real-Life: Chris Torek, Wind River Systems
Intel require I note that my opinions are not those of WRS or Intel
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W)  +1 801 277 2603
email: gmail (figure it out)      http://web.torek.net/torek/index.html

Back to comp.lang.python | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

AW: Py2.7/FreeBSD: maximum number of open files Tobias Oberstein <tobias.oberstein@tavendo.de> - 2011-11-14 09:03 -0800
  Re: Py2.7/FreeBSD: maximum number of open files Jon Clements <joncle@googlemail.com> - 2011-11-14 09:33 -0800
    AW: Py2.7/FreeBSD: maximum number of open files Tobias Oberstein <tobias.oberstein@tavendo.de> - 2011-11-14 09:46 -0800
    Re: Py2.7/FreeBSD: maximum number of open files Christian Heimes <lists@cheimes.de> - 2011-11-14 19:03 +0100
    AW: Py2.7/FreeBSD: maximum number of open files Tobias Oberstein <tobias.oberstein@tavendo.de> - 2011-11-14 10:28 -0800
    Re: Py2.7/FreeBSD: maximum number of open files Christian Heimes <lists@cheimes.de> - 2011-11-14 20:34 +0100
      Re: Py2.7/FreeBSD: maximum number of open files Chris Torek <nospam@torek.net> - 2011-11-30 19:29 +0000

csiph-web