Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
From: Tobias Oberstein <tobias.oberstein@tavendo.de>
To: Christian Heimes <lists@cheimes.de>, "python-list@python.org" <python-list@python.org>
Date: Mon, 14 Nov 2011 10:28:21 -0800
Subject: AW: Py2.7/FreeBSD: maximum number of open files
Thread-Topic: Py2.7/FreeBSD: maximum number of open files
Thread-Index: Acyi+CoZB1UxKu8NRD2cb3wg1vTuiwAAmHTw
References: <634914A010D0B943A035D226786325D42D0C264772@EXVMBX020-12.exch020.serverdata.net> <j9rfda$mfs$1@dough.gmane.org> <634914A010D0B943A035D226786325D42D0C2647C8@EXVMBX020-12.exch020.serverdata.net> <4EC1470B.90403@cheimes.de> <mailman.2705.1321290230.27778.python-list@python.org> <e3901b26-33ab-41b4-a8b1-82d08f15c125@u6g2000vbg.googlegroups.com> <634914A010D0B943A035D226786325D42D0C26481A@EXVMBX020-12.exch020.serverdata.net> <j9rl5k$5i1$1@dough.gmane.org>
In-Reply-To: <j9rl5k$5i1$1@dough.gmane.org>
Accept-Language: de-DE, en-US
Content-Language: de-DE
acceptlanguage: de-DE, en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.2709.1321295306.27778.python-list@python.org>
Lines: 77
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:15686

> > I just confirmed that the bug is even there for FreeBSD 9 RC1 !
> >
> > This is most unfortunate. Seriously.
>=20
> W00t, that sucks! You could migrate to another BSD (NetBSD) or Linux ... =
:)

No, thanks;)

> > I am running out of options, since I am willing to make my stuff
> > Python 3 compatible, but Twisted is not yet there.
> >
> > Using the backported new IO on Python 2.7 will not make open()
> automatically use the new IO, will it?
>=20
> No, the open() function of Python 2.7 will still use the file class which=
 in
> return uses fopen(). You could try to monkey patch the built-in
> open() function. It's mostly API compatible with the current open()
> function:
>=20
>   >>> import io, __builtin__
>   >>> __builtin__.open =3D io.open
>=20
> It works as long as no codes checks for isinstance(obj, file). If your ap=
p only
> has to worry about log files, you might want to overwrite the
> _open() method of logging.FileHandler and its subclasses.
>=20

Thanks! This is probably the most practical option I can go.

I've just tested: the backported new IO on Python 2.7 will indeed
open >32k files on FreeBSD. It also creates the files much faster.
The old, non-monkey-patched version was getting slower and
slower as more files were opened/created ..

There seem to be slight differences though:

Non-monkey patched: I can write to the file a non-Unicode string,
even when the file was opened non-Binary.

With monkey patch: either open the file Binary-mode, or
write Unicode strings ..

I need to see if / what breaks in Twisted.

I can handle my own code .. no problem.

Thanks alot!!



import io, __builtin__
__builtin__.open =3D io.open

import resource

max =3D resource.getrlimit(resource.RLIMIT_NOFILE)
cnt =3D 0
print "maximum FDs", max

max =3D 33000

fds =3D []

while cnt < max:
   f =3D open("/tmp/test1/test_%d" % cnt, "wb")
   f.write("test")
   fds.append(f)
   cnt +=3D 1
   if cnt % 1000 =3D=3D 0:
      print "opened %d files" % cnt

print "ok, created %d files" % cnt