Re: Python garbage collection: not releasing memory to OS!

Path	csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From	Oscar Benjamin <oscar.j.benjamin@gmail.com>
Newsgroups	comp.lang.python
Subject	Re: Python garbage collection: not releasing memory to OS!
Date	Fri, 15 Apr 2016 14:02:30 +0100
Lines	25
Message-ID	<mailman.23.1460725372.6324.python-list@python.org> (permalink)
References	<215cf2a4-c2fa-41d0-8c49-23b9494234d4@googlegroups.com> <CAHVvXxT+Mptniik3PYOqESkqTyHqNDZBqCByxtNevfXf_-k6aA@mail.gmail.com>
Mime-Version	1.0
Content-Type	text/plain; charset=UTF-8
Content-Transfer-Encoding	quoted-printable
X-Trace	news.uni-berlin.de Mqw+hg0TnXke9LermtxcTwkJybfbRBThSRBTJUYk0Wjg==
Return-Path	<oscar.j.benjamin@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.003
X-Spam-Evidence	'H': 0.99; 'S': 0.00; 'subject:Python': 0.05; 'memory.': 0.05; 'bits': 0.07; 'cc:addr:python-list': 0.09; 'spawn': 0.09; 'python': 0.10; 'itself.': 0.11; 'subject:not': 0.11; 'output': 0.13; '2016': 0.16; 'cc:name:python list': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; 'released.': 0.16; 'subprocess': 0.16; 'wrote:': 0.16; 'memory': 0.17; 'config': 0.18; 'input': 0.18; 'cc:2*0': 0.20; 'cc:addr:python.org': 0.20; "aren't": 0.22; 'file.': 0.22; 'finished': 0.23; 'header:In-Reply- To:1': 0.24; 'question': 0.27; 'message-id:@mail.gmail.com': 0.27; 'function': 0.28; 'code': 0.30; 'task': 0.30; 'checked': 0.31; 'problem': 0.33; 'right?': 0.33; 'running': 0.34; 'gets': 0.35; 'received:google.com': 0.35; 'growing': 0.35; 'returning': 0.35; 'skip: 20': 0.35; 'stopped': 0.35; 'asking': 0.35; 'but': 0.36; 'there': 0.36; 'received:209.85': 0.36; 'possible': 0.36; 'subject:: ': 0.37; 'really': 0.37; 'received:209': 0.38; 'does': 0.39; 'your': 0.60; 'real': 0.62; 'here.': 0.62; 'reuse': 0.66; '**is': 0.84; 'oscar': 0.84; 'unclear': 0.84; 'killed': 0.91
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=PhtdXcQZeWM9mYcRFrI6um/PxTUfKjdmeHWo0FlA1RQ=; b=0qstj/JJX7ALablaQdfoBS8Dl6BtbUDLCfvnLc8b9PahXnM5Uf9AZVBLnlPz2Rl+/n Y0YIGfXVo6C/RGWXQ5trOqP/yKzLOypF/fkL7h2kO68tLs6SE8ttJzJqS53Rxo8MiEKs vnEcLHvbPtLl/ubTn0bzNe6cQAwyBk4dGx5ebUZ1eNGQmgup8g6rxWS017cPmUfpAgv6 9DuPhtMK+AuDHNcWg+8KmLYdRLNqVWZ2ziYAXoGE3F4X3O4kU4IP8PBpHw2DEdZKyRRd ON063h902WHHy/NgoPiO0FhddMF8cwR/4TPvmCjgb4cgT+slXTKc3JQDRlfD5VLCZbdB 9iqQ==
X-Google-DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=PhtdXcQZeWM9mYcRFrI6um/PxTUfKjdmeHWo0FlA1RQ=; b=Qmj7Q+j2nLIF3pOvcl8cZ8a5yDYtOEoxVBDqzFQCgBOmtTNwh+FPFJSwSztGtq+UQp GsLGyyQ7HRQqEP/ndxDNQ6Ea6kNrTPC10AjTQ4O6jew/cF7NG+d6kHqu187EabnHRZ+c RKHFCULVaNSs8dCAmrol+aizt6FO2sNYwb3kPjXGawyYtSaIOQmRRQQ/O6QHJeWWlmsC NT88g25mJD++WxwUSG8geKmGeN/6SDJbBQIOEpBDyXgb9w++1Vx9/dVw9B0GnRNX9VLj R5BaAc/mfLstzzer6Sa0RPvdFoOBBdRosroIIWQjj5c43Klfn6wQGOD7jCTZ3zCPAq5A e+9w==
X-Gm-Message-State	AOPr4FXecVH3Ybg5jjpoFlsNGkJ9EpXsBlEw7vBRqScLqEihmXNnSWawB+oES9VfF4300LcEBVziJX651+Pj1w==
X-Received	by 10.25.24.90 with SMTP id o87mr9206377lfi.47.1460725369916; Fri, 15 Apr 2016 06:02:49 -0700 (PDT)
In-Reply-To	<215cf2a4-c2fa-41d0-8c49-23b9494234d4@googlegroups.com>
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.21
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID	<CAHVvXxT+Mptniik3PYOqESkqTyHqNDZBqCByxtNevfXf_-k6aA@mail.gmail.com>
X-Mailman-Original-References	<215cf2a4-c2fa-41d0-8c49-23b9494234d4@googlegroups.com>
Xref	csiph.com comp.lang.python:107055

Show key headers only | View raw

On 15 April 2016 at 11:25,  <cshintov@gmail.com> wrote:
> The input was a 4MB file. Even after returning from the 'fileopen' function the 4MB memory was not released. I checked htop output while the loop was running, the resident memory stays at 14MB. So unless the process is stopped the memory stays with it.

When exactly memory gets freed to the OS is unclear but it's possible
that your process can reuse the same bits of memory. The real question
is whether continuously allocating and deallocating leads to steadily
growing memory usage. If you change it so that your code calls fun
inside the loop you will see that repeatedly calling fun does not lead
to growing memory usage.

> So if the celery worker is not killed after its task is finished it is going to keep the memory for itself. I know I can use **max_tasks_per_child** config value to kill the process and spawn a new one. **Is there any other way to return the memory to OS from a python process?.**

I don't really understand what you're asking here. You're running
celery in a subprocess right? Is the problem about the memory used by
subprocesses that aren't killed or is it the memory usage of the
Python process?

--
Oscar

Thread

Python garbage collection: not releasing memory to OS! cshintov@gmail.com - 2016-04-15 03:25 -0700
  Re: Python garbage collection: not releasing memory to OS! Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2016-04-15 08:24 -0400
  Re: Python garbage collection: not releasing memory to OS! Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2016-04-15 14:02 +0100
  Re: Python garbage collection: not releasing memory to OS! Michael Torrie <torriem@gmail.com> - 2016-04-15 07:59 -0600
  Re: Python garbage collection: not releasing memory to OS! Sam <python@net153.net> - 2016-04-15 09:27 -0500

csiph-web