Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail From: Oscar Benjamin Newsgroups: comp.lang.python Subject: Re: Python garbage collection: not releasing memory to OS! Date: Fri, 15 Apr 2016 14:02:30 +0100 Lines: 25 Message-ID: References: <215cf2a4-c2fa-41d0-8c49-23b9494234d4@googlegroups.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Trace: news.uni-berlin.de Mqw+hg0TnXke9LermtxcTwkJybfbRBThSRBTJUYk0Wjg== Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.003 X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'subject:Python': 0.05; 'memory.': 0.05; 'bits': 0.07; 'cc:addr:python-list': 0.09; 'spawn': 0.09; 'python': 0.10; 'itself.': 0.11; 'subject:not': 0.11; 'output': 0.13; '2016': 0.16; 'cc:name:python list': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; 'released.': 0.16; 'subprocess': 0.16; 'wrote:': 0.16; 'memory': 0.17; 'config': 0.18; 'input': 0.18; 'cc:2**0': 0.20; 'cc:addr:python.org': 0.20; "aren't": 0.22; 'file.': 0.22; 'finished': 0.23; 'header:In-Reply- To:1': 0.24; 'question': 0.27; 'message-id:@mail.gmail.com': 0.27; 'function': 0.28; 'code': 0.30; 'task': 0.30; 'checked': 0.31; 'problem': 0.33; 'right?': 0.33; 'running': 0.34; 'gets': 0.35; 'received:google.com': 0.35; 'growing': 0.35; 'returning': 0.35; 'skip:* 20': 0.35; 'stopped': 0.35; 'asking': 0.35; 'but': 0.36; 'there': 0.36; 'received:209.85': 0.36; 'possible': 0.36; 'subject:: ': 0.37; 'really': 0.37; 'received:209': 0.38; 'does': 0.39; 'your': 0.60; 'real': 0.62; 'here.': 0.62; 'reuse': 0.66; '**is': 0.84; 'oscar': 0.84; 'unclear': 0.84; 'killed': 0.91 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=PhtdXcQZeWM9mYcRFrI6um/PxTUfKjdmeHWo0FlA1RQ=; b=0qstj/JJX7ALablaQdfoBS8Dl6BtbUDLCfvnLc8b9PahXnM5Uf9AZVBLnlPz2Rl+/n Y0YIGfXVo6C/RGWXQ5trOqP/yKzLOypF/fkL7h2kO68tLs6SE8ttJzJqS53Rxo8MiEKs vnEcLHvbPtLl/ubTn0bzNe6cQAwyBk4dGx5ebUZ1eNGQmgup8g6rxWS017cPmUfpAgv6 9DuPhtMK+AuDHNcWg+8KmLYdRLNqVWZ2ziYAXoGE3F4X3O4kU4IP8PBpHw2DEdZKyRRd ON063h902WHHy/NgoPiO0FhddMF8cwR/4TPvmCjgb4cgT+slXTKc3JQDRlfD5VLCZbdB 9iqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=PhtdXcQZeWM9mYcRFrI6um/PxTUfKjdmeHWo0FlA1RQ=; b=Qmj7Q+j2nLIF3pOvcl8cZ8a5yDYtOEoxVBDqzFQCgBOmtTNwh+FPFJSwSztGtq+UQp GsLGyyQ7HRQqEP/ndxDNQ6Ea6kNrTPC10AjTQ4O6jew/cF7NG+d6kHqu187EabnHRZ+c RKHFCULVaNSs8dCAmrol+aizt6FO2sNYwb3kPjXGawyYtSaIOQmRRQQ/O6QHJeWWlmsC NT88g25mJD++WxwUSG8geKmGeN/6SDJbBQIOEpBDyXgb9w++1Vx9/dVw9B0GnRNX9VLj R5BaAc/mfLstzzer6Sa0RPvdFoOBBdRosroIIWQjj5c43Klfn6wQGOD7jCTZ3zCPAq5A e+9w== X-Gm-Message-State: AOPr4FXecVH3Ybg5jjpoFlsNGkJ9EpXsBlEw7vBRqScLqEihmXNnSWawB+oES9VfF4300LcEBVziJX651+Pj1w== X-Received: by 10.25.24.90 with SMTP id o87mr9206377lfi.47.1460725369916; Fri, 15 Apr 2016 06:02:49 -0700 (PDT) In-Reply-To: <215cf2a4-c2fa-41d0-8c49-23b9494234d4@googlegroups.com> X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Mailman-Original-Message-ID: X-Mailman-Original-References: <215cf2a4-c2fa-41d0-8c49-23b9494234d4@googlegroups.com> Xref: csiph.com comp.lang.python:107055 On 15 April 2016 at 11:25, wrote: > The input was a 4MB file. Even after returning from the 'fileopen' functi= on the 4MB memory was not released. I checked htop output while the loop wa= s running, the resident memory stays at 14MB. So unless the process is stop= ped the memory stays with it. When exactly memory gets freed to the OS is unclear but it's possible that your process can reuse the same bits of memory. The real question is whether continuously allocating and deallocating leads to steadily growing memory usage. If you change it so that your code calls fun inside the loop you will see that repeatedly calling fun does not lead to growing memory usage. > So if the celery worker is not killed after its task is finished it is go= ing to keep the memory for itself. I know I can use **max_tasks_per_child**= config value to kill the process and spawn a new one. **Is there any other= way to return the memory to OS from a python process?.** I don't really understand what you're asking here. You're running celery in a subprocess right? Is the problem about the memory used by subprocesses that aren't killed or is it the memory usage of the Python process? -- Oscar