Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed3a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
From: Nick Timkovich <prometheus235@gmail.com>
Date: Fri, 14 Feb 2014 16:27:58 -0600
Subject: Generator using item[n-1] + item[n] memory
To: python-list@python.org
Content-Type: multipart/alternative; boundary=089e01493cae9976fb04f265526b
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.6941.1392416906.18130.python-list@python.org>
Lines: 209
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:66335

--089e01493cae9976fb04f265526b
Content-Type: text/plain; charset=ISO-8859-1

I have a Python 3.x program that processes several large text files that
contain sizeable arrays of data that can occasionally brush up against the
memory limit of my puny workstation.  From some basic memory profiling, it
seems like when using the generator, the memory usage of my script balloons
to hold consecutive elements, using up to twice the memory I expect.

I made a simple, stand alone example to test the generator and I get
similar results in Python 2.7, 3.3, and 3.4.  My test code follows,
`memory_usage()` is a modifed version of [this function from an SO
question](http://stackoverflow.com/a/898406/194586) which uses
`/proc/self/status` and agrees with `top` as I watch it.  `resource` is
probably a more cross-platform method:

###############

import sys, resource, gc, time

def biggen():
    sizes = 1, 1, 10, 1, 1, 10, 10, 1, 1, 10, 10, 20, 1, 1, 20, 20, 1, 1
    for size in sizes:
        data = [1] * int(size * 1e6)
        #time.sleep(1)
        yield data

def consumer():
    for data in biggen():
        rusage = resource.getrusage(resource.RUSAGE_SELF)
        peak_mb = rusage.ru_maxrss/1024.0
        print('Peak: {0:6.1f} MB, Data Len: {1:6.1f} M'.format(
                peak_mb, len(data)/1e6))
        #print(memory_usage())

        data = None  # go
        del data     # away
        gc.collect() # please.

# def memory_usage():
#     """Memory usage of the current process, requires /proc/self/status"""
#     # http://stackoverflow.com/a/898406/194586
#     result = {'peak': 0, 'rss': 0}
#     for line in open('/proc/self/status'):
#         parts = line.split()
#         key = parts[0][2:-1].lower()
#         if key in result:
#             result[key] = int(parts[1])/1024.0
#     return 'Peak: {peak:6.1f} MB, Current: {rss:6.1f} MB'.format(**result)

print(sys.version)
consumer()

###############

In practice I'll process data coming from such a generator loop, saving
just what I need, then discard it.

When I run the above script, and two large elements come in series (the
data size can be highly variable), it seems like Python computes the next
before freeing the previous, leading to up to double the memory usage.

    $ python genmem.py
    2.7.3 (default, Sep 26 2013, 20:08:41)
    [GCC 4.6.3]
    Peak:    7.9 MB, Data Len:    1.0 M
    Peak:   11.5 MB, Data Len:    1.0 M
    Peak:   45.8 MB, Data Len:   10.0 M
    Peak:   45.9 MB, Data Len:    1.0 M
    Peak:   45.9 MB, Data Len:    1.0 M
    Peak:   45.9 MB, Data Len:   10.0 M
    #        ^^  not much different versus previous 10M-list
    Peak:   80.2 MB, Data Len:   10.0 M
    #        ^^  same list size, but new memory peak at roughly twice the
usage
    Peak:   80.2 MB, Data Len:    1.0 M
    Peak:   80.2 MB, Data Len:    1.0 M
    Peak:   80.2 MB, Data Len:   10.0 M
    Peak:   80.2 MB, Data Len:   10.0 M
    Peak:  118.3 MB, Data Len:   20.0 M
    #        ^^  and again...  (20+10)*c
    Peak:  118.3 MB, Data Len:    1.0 M
    Peak:  118.3 MB, Data Len:    1.0 M
    Peak:  118.3 MB, Data Len:   20.0 M
    Peak:  156.5 MB, Data Len:   20.0 M
    #        ^^  and again. (20+20)*c
    Peak:  156.5 MB, Data Len:    1.0 M
    Peak:  156.5 MB, Data Len:    1.0 M

The crazy belt-and-suspenders-and-duct-tape approach `data = None`, `del
data`, and `gc.collect()` does nothing.

I'm pretty sure the generator itself is not doubling up on memory because
otherwise a single large value it yields would increase the peak usage, and
in the *same iteration* a large object appeared; it's only large
consecutive objects.

How can I save my memory?

Cheers,
Nick

cc: StackOverflow http://stackoverflow.com/q/21787099/194586

--089e01493cae9976fb04f265526b
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>I have a Python 3.x program that processes several la=
rge text files that contain sizeable arrays of data that can occasionally b=
rush up against the memory limit of my puny workstation. =A0From some basic=
 memory profiling, it seems like when using the generator, the memory usage=
 of my script balloons to hold consecutive elements, using up to twice the =
memory I expect.</div>

<div><br></div><div>I made a simple, stand alone example to test the genera=
tor and I get similar results in Python 2.7, 3.3, and 3.4. =A0My test code =
follows, `memory_usage()` is a modifed version of [this function from an SO=
 question](<a href=3D"http://stackoverflow.com/a/898406/194586">http://stac=
koverflow.com/a/898406/194586</a>) which uses `/proc/self/status` and agree=
s with `top` as I watch it. =A0`resource` is probably a more cross-platform=
 method:</div>

<div><br></div><div><div>###############</div></div><div><br></div><div><di=
v>import sys, resource, gc, time</div><div><br></div><div>def biggen():</di=
v><div>=A0 =A0 sizes =3D 1, 1, 10, 1, 1, 10, 10, 1, 1, 10, 10, 20, 1, 1, 20=
, 20, 1, 1</div>

<div>=A0 =A0 for size in sizes:</div><div>=A0 =A0 =A0 =A0 data =3D [1] * in=
t(size * 1e6)</div><div>=A0 =A0 =A0 =A0 #time.sleep(1)</div><div>=A0 =A0 =
=A0 =A0 yield data</div><div><br></div><div>def consumer():</div><div>=A0 =
=A0 for data in biggen():</div><div>

=A0 =A0 =A0 =A0 rusage =3D resource.getrusage(resource.RUSAGE_SELF)</div><d=
iv>=A0 =A0 =A0 =A0 peak_mb =3D rusage.ru_maxrss/1024.0</div><div>=A0 =A0 =
=A0 =A0 print(&#39;Peak: {0:6.1f} MB, Data Len: {1:6.1f} M&#39;.format(</di=
v><div>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 peak_mb, len(data)/1e6))</div>

<div>=A0 =A0 =A0 =A0 #print(memory_usage())</div><div><br></div><div>=A0 =
=A0 =A0 =A0 data =3D None =A0# go</div><div>=A0 =A0 =A0 =A0 del data =A0 =
=A0 # away</div><div>=A0 =A0 =A0 =A0 gc.collect() # please.</div><div><br><=
/div><div># def memory_usage():</div><div>

# =A0 =A0 &quot;&quot;&quot;Memory usage of the current process, requires /=
proc/self/status&quot;&quot;&quot;</div><div># =A0 =A0 # <a href=3D"http://=
stackoverflow.com/a/898406/194586">http://stackoverflow.com/a/898406/194586=
</a></div>

<div># =A0 =A0 result =3D {&#39;peak&#39;: 0, &#39;rss&#39;: 0}</div><div>#=
 =A0 =A0 for line in open(&#39;/proc/self/status&#39;):</div><div># =A0 =A0=
 =A0 =A0 parts =3D line.split()</div><div># =A0 =A0 =A0 =A0 key =3D parts[0=
][2:-1].lower()</div><div>

# =A0 =A0 =A0 =A0 if key in result:</div><div># =A0 =A0 =A0 =A0 =A0 =A0 res=
ult[key] =3D int(parts[1])/1024.0</div><div># =A0 =A0 return &#39;Peak: {pe=
ak:6.1f} MB, Current: {rss:6.1f} MB&#39;.format(**result)</div><div><br></d=
iv><div>print(sys.version)</div>

<div>consumer()</div></div><div><br></div><div>###############</div><div><b=
r></div><div>In practice I&#39;ll process data coming from such a generator=
 loop, saving just what I need, then discard it.</div><div><br></div><div>

When I run the above script, and two large elements come in series (the dat=
a size can be highly variable), it seems like Python computes the next befo=
re freeing the previous, leading to up to double the memory usage.</div>

<div><br></div><div>=A0 =A0 $ python genmem.py=A0</div><div>=A0 =A0 2.7.3 (=
default, Sep 26 2013, 20:08:41)=A0</div><div>=A0 =A0 [GCC 4.6.3]</div><div>=
=A0 =A0 Peak: =A0 =A07.9 MB, Data Len: =A0 =A01.0 M</div><div>=A0 =A0 Peak:=
 =A0 11.5 MB, Data Len: =A0 =A01.0 M</div>

<div>=A0 =A0 Peak: =A0 45.8 MB, Data Len: =A0 10.0 M</div><div>=A0 =A0 Peak=
: =A0 45.9 MB, Data Len: =A0 =A01.0 M</div><div>=A0 =A0 Peak: =A0 45.9 MB, =
Data Len: =A0 =A01.0 M</div><div>=A0 =A0 Peak: =A0 45.9 MB, Data Len: =A0 1=
0.0 M</div><div>=A0 =A0 # =A0 =A0 =A0 =A0^^ =A0not much different versus pr=
evious 10M-list</div>

<div>=A0 =A0 Peak: =A0 80.2 MB, Data Len: =A0 10.0 M</div><div>=A0 =A0 # =
=A0 =A0 =A0 =A0^^ =A0same list size, but new memory peak at roughly twice t=
he usage</div><div>=A0 =A0 Peak: =A0 80.2 MB, Data Len: =A0 =A01.0 M</div><=
div>=A0 =A0 Peak: =A0 80.2 MB, Data Len: =A0 =A01.0 M</div>

<div>=A0 =A0 Peak: =A0 80.2 MB, Data Len: =A0 10.0 M</div><div>=A0 =A0 Peak=
: =A0 80.2 MB, Data Len: =A0 10.0 M</div><div>=A0 =A0 Peak: =A0118.3 MB, Da=
ta Len: =A0 20.0 M</div><div>=A0 =A0 # =A0 =A0 =A0 =A0^^ =A0and again... =
=A0(20+10)*c</div><div>=A0 =A0 Peak: =A0118.3 MB, Data Len: =A0 =A01.0 M</d=
iv>

<div>=A0 =A0 Peak: =A0118.3 MB, Data Len: =A0 =A01.0 M</div><div>=A0 =A0 Pe=
ak: =A0118.3 MB, Data Len: =A0 20.0 M</div><div>=A0 =A0 Peak: =A0156.5 MB, =
Data Len: =A0 20.0 M</div><div>=A0 =A0 # =A0 =A0 =A0 =A0^^ =A0and again. (2=
0+20)*c</div><div>=A0 =A0 Peak: =A0156.5 MB, Data Len: =A0 =A01.0 M</div>

<div>=A0 =A0 Peak: =A0156.5 MB, Data Len: =A0 =A01.0 M</div><div><br></div>=
<div>The crazy belt-and-suspenders-and-duct-tape approach `data =3D None`, =
`del data`, and `gc.collect()` does nothing.</div><div><br></div><div>I&#39=
;m pretty sure the generator itself is not doubling up on memory because ot=
herwise a single large value it yields would increase the peak usage, and i=
n the *same iteration* a large object appeared; it&#39;s only large consecu=
tive objects.</div>

<div><br></div><div>How can I save my memory?</div><div><br></div><div>Chee=
rs,</div><div>Nick</div><div><br></div><div>cc: StackOverflow=A0<a href=3D"=
http://stackoverflow.com/q/21787099/194586">http://stackoverflow.com/q/2178=
7099/194586</a></div>

</div>

--089e01493cae9976fb04f265526b--