Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.lang.python > #78844

Re: "High water" Memory fragmentation still a thing?

X-Received by 10.66.66.196 with SMTP id h4mr7354429pat.18.1412789320065; Wed, 08 Oct 2014 10:28:40 -0700 (PDT)
X-Received by 10.140.95.113 with SMTP id h104mr23778qge.36.1412789320012; Wed, 08 Oct 2014 10:28:40 -0700 (PDT)
Path csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!news.glorb.com!uq10no6593236igb.0!news-out.google.com!q8ni67qal.1!nntp.google.com!dc16no1484575qab.1!postnews.google.com!glegroupsg2000goo.googlegroups.com!not-for-mail
Newsgroups comp.lang.python
Date Wed, 8 Oct 2014 10:28:39 -0700 (PDT)
In-Reply-To <mailman.14647.1412613796.18130.python-list@python.org>
Complaints-To groups-abuse@google.com
Injection-Info glegroupsg2000goo.googlegroups.com; posting-host=108.91.71.236; posting-account=J5j24gkAAABQ9Yx9iE8ZAnhHMzpxizi7
NNTP-Posting-Host 108.91.71.236
References <CANm61jc=pB7n+5kuea6bmYXTbo3hXLT8d4JZx3Yq-TBtgGrFOQ@mail.gmail.com> <mailman.14590.1412378803.18130.python-list@python.org> <7ac186c3-2dbc-4cfa-ab58-37854aeb59d1@googlegroups.com> <mailman.14647.1412613796.18130.python-list@python.org>
User-Agent G2/1.0
MIME-Version 1.0
Message-ID <b5ce5cc2-3a37-49e4-aa2a-d29dd3dda972@googlegroups.com> (permalink)
Subject Re: "High water" Memory fragmentation still a thing?
From bryanjugglercryptographer@yahoo.com
Injection-Date Wed, 08 Oct 2014 17:28:40 +0000
Content-Type text/plain; charset=ISO-8859-1
Content-Transfer-Encoding quoted-printable
Xref csiph.com comp.lang.python:78844

Show key headers only | View raw


Chris Angelico wrote:
> Sure, and that's all well and good. But what I just cited there *is* a
> shipping product. That's a live server that runs a game that I'm admin
> of. So it's possible to do without the resource safety net of periodic
> restarts.

Nice that the non-Python server you administer stayed up for 88 weeks, but that doesn't really have anything to do with the issue here. The answer to the OP's title question is "yes", high-water memory fragmentation is a real thing, in most platforms including CPython.

The cited article tells of Celery hitting the problem, and the working solution was to "roll the celery worker processes". That doesn't mean to tell a human administrator to regularly restart the server. It's programmatic and it's a reasonably simple and well-established design pattern.

> > For an example see the Apache HTTP daemon, particularly the classic pre-forking server. There's a configuration parameter, "MaxRequestsPerChild", that sets how many requests a process should answer before terminating.
> 
> That assumes that requests can be handled equally by any server
> process - and more so, that there are such things as discrete
> requests. That's true of HTTP, but not of everything. 

It's true of HTTP and many other protocols because they were designed to support robust operation even as individual components may fail.

> And even with
> HTTP, if you do "long polls" [1] then clients might remain connected
> for arbitrary lengths of time; either you have to cut them off when
> you terminate the server process (hopefully that isn't too often, or
> you lose the benefit of long polling), or you retain processes for
> much longer.

If you look at actual long-polling protocols, you'll see that the server occasionally closing connections is no problem at all. They're actually designed to be robust even against connections that drop without proper shutdown.

> Restarting isn't necessary. It's like rebooting a computer: people get
> into the habit of doing it, because it "fixes problems", but all that
> means is that it allows you to get sloppy with resource management.

CPython, and for that matter malloc/free, have known problems in resource management, such as the fragmentation issue noted here. There are more. Try a Google site search for "memory leak" on http://bugs.python.org/. Do you think the last memory leak is fixed now?

From what I've seen, planned process replacement is the primary techniques to support long-lived mission-critical services in face of resource management flaws. Upon process termination the OS recovers the resources. I love CPython, but on this point I trust the Linux kernel much more.

-- 
--Bryan

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread


Thread

Re: "High water" Memory fragmentation still a thing? Chris Angelico <rosuav@gmail.com> - 2014-10-04 09:26 +1000
  Re: "High water" Memory fragmentation still a thing? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-10-04 11:02 +1000
    Re: "High water" Memory fragmentation still a thing? Chris Angelico <rosuav@gmail.com> - 2014-10-04 12:22 +1000
      Re: "High water" Memory fragmentation still a thing? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-10-04 15:48 +1000
        Re: "High water" Memory fragmentation still a thing? Chris Angelico <rosuav@gmail.com> - 2014-10-04 15:57 +1000
        Re: "High water" Memory fragmentation still a thing? Sturla Molden <sturla.molden@gmail.com> - 2014-10-04 17:51 +0000
    Re: "High water" Memory fragmentation still a thing? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-10-04 16:20 -0400
    Re: "High water" Memory fragmentation still a thing? mm0fmf <none@mailinator.com> - 2014-10-05 20:23 +0100
      Re: "High water" Memory fragmentation still a thing? Steven D'Aprano <steve@pearwood.info> - 2014-10-06 07:20 +0000
  Re: "High water" Memory fragmentation still a thing? bryanjugglercryptographer@yahoo.com - 2014-10-06 09:21 -0700
    Re: "High water" Memory fragmentation still a thing? Chris Angelico <rosuav@gmail.com> - 2014-10-07 03:43 +1100
      Re: "High water" Memory fragmentation still a thing? bryanjugglercryptographer@yahoo.com - 2014-10-08 10:28 -0700
        Re: "High water" Memory fragmentation still a thing? Emile van Sebille <emile@fenx.com> - 2014-10-09 14:39 -0700
        Re: "High water" Memory fragmentation still a thing? Chris Angelico <rosuav@gmail.com> - 2014-10-10 08:42 +1100
        Re: "High water" Memory fragmentation still a thing? dieter <dieter@handshake.de> - 2014-10-10 08:09 +0200
        Re: "High water" Memory fragmentation still a thing? Chris Angelico <rosuav@gmail.com> - 2014-10-10 17:52 +1100

csiph-web