Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!goblin2!goblin.stu.neva.ru!newsfeed.xs4all.nl!newsfeed3a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <CANm61jc=pB7n+5kuea6bmYXTbo3hXLT8d4JZx3Yq-TBtgGrFOQ@mail.gmail.com>
References: <CANm61jc=pB7n+5kuea6bmYXTbo3hXLT8d4JZx3Yq-TBtgGrFOQ@mail.gmail.com>
Date: Sat, 4 Oct 2014 09:26:40 +1000
Subject: Re: "High water" Memory fragmentation still a thing?
From: Chris Angelico <rosuav@gmail.com>
Cc: "python-list@python.org" <python-list@python.org>
Content-Type: text/plain; charset=UTF-8
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.14590.1412378803.18130.python-list@python.org>
Lines: 31
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:78673

On Sat, Oct 4, 2014 at 4:36 AM, Croepha <croepha@gmail.com> wrote:
> What this means is that processes that do need to use a lot of memory will
> exhibit a "high water" behavior, where they remain forever at the level of
> memory usage that they required at their peak.

This is almost never true. What you will see, though, is something
like Christian described; pages get allocated, and then partially
used, and you don't always get all that memory back. In theory, a high
level language like Python would be allowed to move objects around to
compact memory, but CPython doesn't do this, and there's no proof that
it'd really help anything anyway. (Look at Christian's comments about
"Should you return it or not?" and the cost of system calls... now
consider the orders-of-magnitude worse cost of actually moving memory
around.)

This is why a lot of long-duration processes are built to be restarted
periodically. It's not strictly necessary, but it can be the most
effective way of solving a problem. I tend to ignore that, though, and
let my processes just keep on running... for 88 wk 4d 23:56:27 so far,
on one of those processes. It's consuming less than half a gig of
virtual memory, quarter gig resident, and it's been doing a fair bit
(it keeps all sorts of things in memory to avoid re-decoding from
disk). So don't worry too much about memory usage until you see that
there's actually a problem; with most Python processes, you'll restart
them to deploy new code sooner than you'll restart them to fix memory
problems. (The above example isn't a Python process, and code updates
happen live.) In fact, I'd advise that as a general policy: Don't
panic about any Python limitation until you've proven that it's
actually a problem. :)

ChrisA