Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #15686

AW: Py2.7/FreeBSD: maximum number of open files

Show all headers | View raw

> > I just confirmed that the bug is even there for FreeBSD 9 RC1 !
> >
> > This is most unfortunate. Seriously.
> 
> W00t, that sucks! You could migrate to another BSD (NetBSD) or Linux ... :)

No, thanks;)

> > I am running out of options, since I am willing to make my stuff
> > Python 3 compatible, but Twisted is not yet there.
> >
> > Using the backported new IO on Python 2.7 will not make open()
> automatically use the new IO, will it?
> 
> No, the open() function of Python 2.7 will still use the file class which in
> return uses fopen(). You could try to monkey patch the built-in
> open() function. It's mostly API compatible with the current open()
> function:
> 
>   >>> import io, __builtin__
>   >>> __builtin__.open = io.open
> 
> It works as long as no codes checks for isinstance(obj, file). If your app only
> has to worry about log files, you might want to overwrite the
> _open() method of logging.FileHandler and its subclasses.
> 

Thanks! This is probably the most practical option I can go.

I've just tested: the backported new IO on Python 2.7 will indeed
open >32k files on FreeBSD. It also creates the files much faster.
The old, non-monkey-patched version was getting slower and
slower as more files were opened/created ..

There seem to be slight differences though:

Non-monkey patched: I can write to the file a non-Unicode string,
even when the file was opened non-Binary.

With monkey patch: either open the file Binary-mode, or
write Unicode strings ..

I need to see if / what breaks in Twisted.

I can handle my own code .. no problem.

Thanks alot!!



import io, __builtin__
__builtin__.open = io.open

import resource

max = resource.getrlimit(resource.RLIMIT_NOFILE)
cnt = 0
print "maximum FDs", max

max = 33000

fds = []

while cnt < max:
   f = open("/tmp/test1/test_%d" % cnt, "wb")
   f.write("test")
   fds.append(f)
   cnt += 1
   if cnt % 1000 == 0:
      print "opened %d files" % cnt

print "ok, created %d files" % cnt

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

AW: Py2.7/FreeBSD: maximum number of open files Tobias Oberstein <tobias.oberstein@tavendo.de> - 2011-11-14 09:03 -0800
  Re: Py2.7/FreeBSD: maximum number of open files Jon Clements <joncle@googlemail.com> - 2011-11-14 09:33 -0800
    AW: Py2.7/FreeBSD: maximum number of open files Tobias Oberstein <tobias.oberstein@tavendo.de> - 2011-11-14 09:46 -0800
    Re: Py2.7/FreeBSD: maximum number of open files Christian Heimes <lists@cheimes.de> - 2011-11-14 19:03 +0100
    AW: Py2.7/FreeBSD: maximum number of open files Tobias Oberstein <tobias.oberstein@tavendo.de> - 2011-11-14 10:28 -0800
    Re: Py2.7/FreeBSD: maximum number of open files Christian Heimes <lists@cheimes.de> - 2011-11-14 20:34 +0100
      Re: Py2.7/FreeBSD: maximum number of open files Chris Torek <nospam@torek.net> - 2011-11-30 19:29 +0000

csiph-web