Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #44665

Trying to understand the memory occupation of big lists

Show all headers | View raw

I have a memory leak in a program using big arrays. With the goal of debugging it I run into the memory_profiler module. Then I discovered something which is surprising to me. Please consider the following script:

$ cat memtest.py
import gc
from memory_profiler import profile


@profile
def test1():
    a = [0] * 1024 * 1024
    del a
    gc.collect()  # nothing change if I comment this


@profile
def test2():
    for i in range(10):
        a = [0] * 1024 * 1024
        del a
    gc.collect()  # nothing change if I comment this


test1()
test2()

Here is its output, on a Linux 64 bit machine:

$ python memtest.py
Filename: memtest.py

Line #    Mem usage    Increment   Line Contents
================================================
     5                             @profile
     6     9.250 MB     0.000 MB   def test1():
     7    17.246 MB     7.996 MB       a = [0] * 1024 * 1024
     8     9.258 MB    -7.988 MB       del a
     9     9.258 MB     0.000 MB       gc.collect()  # nothing change if I comment this


Filename: memtest.py

Line #    Mem usage    Increment   Line Contents
================================================
    12                             @profile
    13     9.262 MB     0.000 MB   def test2():
    14    17.270 MB     8.008 MB       for i in range(10):
    15    17.270 MB     0.000 MB           a = [0] * 1024 * 1024
    16    17.270 MB     0.000 MB           del a
    17    17.270 MB     0.000 MB       gc.collect()  # nothing change if I comment this

In the first case the memory is released (even if strangely not
completely, 7.996 != 7.988), in the second case the memory is not. Why it is so? I did expect gc.collect() to free the memory but it is completely ininfluent. In the second cases there are 10 lists with 8 MB each, so
80 MB are allocated and 72 released, but 8 MB are still there apparently.
It does not look like a problem of mem_profile, this is what observe with
top too.

Any ideas?

Back to comp.lang.python | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

Trying to understand the memory occupation of big lists Michele Simionato <michele.simionato@gmail.com> - 2013-05-03 04:24 -0700
  Re: Trying to understand the memory occupation of big lists Dave Angel <davea@davea.name> - 2013-05-03 08:16 -0400
  Re: Trying to understand the memory occupation of big lists Maarten <maarten.sneep@knmi.nl> - 2013-05-03 08:20 -0700

csiph-web