Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #44665 > unrolled thread

Trying to understand the memory occupation of big lists

Back to article view | Back to comp.lang.python

Contents

  Trying to understand the memory occupation of big lists Michele Simionato <michele.simionato@gmail.com> - 2013-05-03 04:24 -0700
    Re: Trying to understand the memory occupation of big lists Dave Angel <davea@davea.name> - 2013-05-03 08:16 -0400
    Re: Trying to understand the memory occupation of big lists Maarten <maarten.sneep@knmi.nl> - 2013-05-03 08:20 -0700

#44665 — Trying to understand the memory occupation of big lists

I have a memory leak in a program using big arrays. With the goal of debugging it I run into the memory_profiler module. Then I discovered something which is surprising to me. Please consider the following script:

$ cat memtest.py
import gc
from memory_profiler import profile


@profile
def test1():
    a = [0] * 1024 * 1024
    del a
    gc.collect()  # nothing change if I comment this


@profile
def test2():
    for i in range(10):
        a = [0] * 1024 * 1024
        del a
    gc.collect()  # nothing change if I comment this


test1()
test2()

Here is its output, on a Linux 64 bit machine:

$ python memtest.py
Filename: memtest.py

Line #    Mem usage    Increment   Line Contents
================================================
     5                             @profile
     6     9.250 MB     0.000 MB   def test1():
     7    17.246 MB     7.996 MB       a = [0] * 1024 * 1024
     8     9.258 MB    -7.988 MB       del a
     9     9.258 MB     0.000 MB       gc.collect()  # nothing change if I comment this


Filename: memtest.py

Line #    Mem usage    Increment   Line Contents
================================================
    12                             @profile
    13     9.262 MB     0.000 MB   def test2():
    14    17.270 MB     8.008 MB       for i in range(10):
    15    17.270 MB     0.000 MB           a = [0] * 1024 * 1024
    16    17.270 MB     0.000 MB           del a
    17    17.270 MB     0.000 MB       gc.collect()  # nothing change if I comment this

In the first case the memory is released (even if strangely not
completely, 7.996 != 7.988), in the second case the memory is not. Why it is so? I did expect gc.collect() to free the memory but it is completely ininfluent. In the second cases there are 10 lists with 8 MB each, so
80 MB are allocated and 72 released, but 8 MB are still there apparently.
It does not look like a problem of mem_profile, this is what observe with
top too.

Any ideas?

[toc] | [next] | [standalone]

#44666

On 05/03/2013 07:24 AM, Michele Simionato wrote:
> I have a memory leak in a program using big arrays.

Actually, big lists.  Python also has arrays, and they're entirely 
different.

With the goal of debugging it I run into the memory_profiler module. 
Then I discovered something which is surprising to me. Please consider 
the following script:
>
> $ cat memtest.py
> import gc
> from memory_profiler import profile
>
>
> @profile
> def test1():
>      a = [0] * 1024 * 1024
>      del a
>      gc.collect()  # nothing change if I comment this
>
>
> @profile
> def test2():
>      for i in range(10):
>          a = [0] * 1024 * 1024
>          del a
>      gc.collect()  # nothing change if I comment this
>
>
> test1()
> test2()
>
> Here is its output, on a Linux 64 bit machine:
>
> $ python memtest.py
> Filename: memtest.py
>
> Line #    Mem usage    Increment   Line Contents
> ================================================
>       5                             @profile
>       6     9.250 MB     0.000 MB   def test1():
>       7    17.246 MB     7.996 MB       a = [0] * 1024 * 1024
>       8     9.258 MB    -7.988 MB       del a
>       9     9.258 MB     0.000 MB       gc.collect()  # nothing change if I comment this
>
>
> Filename: memtest.py
>
> Line #    Mem usage    Increment   Line Contents
> ================================================
>      12                             @profile
>      13     9.262 MB     0.000 MB   def test2():
>      14    17.270 MB     8.008 MB       for i in range(10):
>      15    17.270 MB     0.000 MB           a = [0] * 1024 * 1024
>      16    17.270 MB     0.000 MB           del a
>      17    17.270 MB     0.000 MB       gc.collect()  # nothing change if I comment this
>
> In the first case the memory is released (even if strangely not
> completely, 7.996 != 7.988), in the second case the memory is not. Why it is so? I did expect gc.collect() to free the memory but it is completely ininfluent. In the second cases there are 10 lists with 8 MB each, so
> 80 MB are allocated and 72 released, but 8 MB are still there apparently.
> It does not look like a problem of mem_profile, this is what observe with
> top too.
>
> Any ideas?
>

I haven't played with profile, so my comments are limited to the direct 
code.

gd.collect() has nothing to do in either of these functions, since the 
memory has already been released by the ref-count logic.  Only in the 
case of a circular reference is the gc.collect() useful.  If you want to 
see gc.collect() in action create two large objects that reference each 
other and a small one that references one of them.  Del the first two 
and then the third, and the memory cannot be released since the ref 
counts are nonzero.  Then do a gc.collect() which will realize that you 
have no way to reference either of the two large objects.

I suspect that profile is only looking at the memory from the point of 
view of the OS.  No block of memory can be released to the OS unless 
it's entirely freed.  My guess is that in the second case the variable i 
(or some other internal one relating to the loop) is in the same block 
with one of those lists.  The point is that CPython uses the C malloc() 
and free() functions, and they have their own limitations.  Most of the 
time when free() is called, the memory is NOT released to the OS, but is 
still made available within Python for future use.


-- 
DaveA

[toc] | [prev] | [next] | [standalone]

#44669

I made a few changes:

import gc
from memory_profiler import profile

@profile
def test1():
    a = [0] * 1024**2
    del a
    a = [0] * 1024**2
    del a
    a = [0] * 1024**2
    del a
    a = [0] * 1024**2
    del a
    a = [0] * 1024**2
    del a
    a = [0] * 1024**2
    del a
    a = [0] * 1024**2
    del a
    a = [0] * 1024**2
    del a
    a = [0] * 1024**2
    del a
    a = [0] * 1024**2
    del a
    gc.collect()  # nothing change if I comment this


@profile
def test2():
    for i in range(10):
        a = [0] * 1024**2
        del a
    del i
    gc.collect()  # nothing change if I comment this


test1()
test2() 

# end of code

Output:

Filename: profile.py

Line #    Mem usage    Increment   Line Contents
================================================
     5                             @profile
     6     8.688 MB     0.000 MB   def test1():
     7    16.691 MB     8.004 MB       a = [0] * 1024**2
     8     8.688 MB    -8.004 MB       del a
     9    16.680 MB     7.992 MB       a = [0] * 1024**2
    10    16.680 MB     0.000 MB       del a
    11    16.680 MB     0.000 MB       a = [0] * 1024**2
    12    16.680 MB     0.000 MB       del a
    13    16.680 MB     0.000 MB       a = [0] * 1024**2
    14    16.680 MB     0.000 MB       del a
    15    16.680 MB     0.000 MB       a = [0] * 1024**2
    16    16.680 MB     0.000 MB       del a
    17    16.680 MB     0.000 MB       a = [0] * 1024**2
    18    16.680 MB     0.000 MB       del a
    19    16.680 MB     0.000 MB       a = [0] * 1024**2
    20    16.680 MB     0.000 MB       del a
    21    16.680 MB     0.000 MB       a = [0] * 1024**2
    22    16.680 MB     0.000 MB       del a
    23    16.680 MB     0.000 MB       a = [0] * 1024**2
    24    16.680 MB     0.000 MB       del a
    25    16.680 MB     0.000 MB       a = [0] * 1024**2
    26    16.680 MB     0.000 MB       del a
    27    16.680 MB     0.000 MB       gc.collect()  # nothing change if I comment this


Filename: profile.py

Line #    Mem usage    Increment   Line Contents
================================================
    30                             @profile
    31    16.691 MB     0.000 MB   def test2():
    32    16.691 MB     0.000 MB       for i in range(10):
    33    16.691 MB     0.000 MB           a = [0] * 1024**2
    34    16.691 MB     0.000 MB           del a
    35    16.691 MB     0.000 MB       del i
    36    16.691 MB     0.000 MB       gc.collect()  # nothing change if I comment this

If I make the two functions identical, the behave the same.

Maarten

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web