Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!newsfeed.xs4all.nl!newsfeed2a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <87d2hyrkf4.fsf@elektro.pacujo.net>
References: <CACwCsY65gpn_Zx78wquE-rmrZu10pu9GYbzWd-Y3ypzHsMb2Yw@mail.gmail.com> <CACwCsY4nxs-4WmKboA9mOwrjQjXKXmaBPTuuq0GSmjzEpA3efQ@mail.gmail.com> <CAPTjJmr=1CW3BP23x2RTmnfxWppUNz_m_jHp=6aBTF0qj3YkVw@mail.gmail.com> <CACwCsY4LCg5Mw-fx_Og93ErJ_N14Vqw7TO2TrEnp3wAdY0POkg@mail.gmail.com> <mailman.7879.1394144925.18130.python-list@python.org> <87ha7bq7ls.fsf@elektro.pacujo.net> <mailman.7880.1394145824.18130.python-list@python.org> <87d2hyrkf4.fsf@elektro.pacujo.net>
Date: Fri, 7 Mar 2014 10:31:46 +1100
Subject: Re: script uses up all memory
From: Chris Angelico <rosuav@gmail.com>
Cc: "python-list@python.org" <python-list@python.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.7881.1394148716.18130.python-list@python.org>
Lines: 91
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:67963

On Fri, Mar 7, 2014 at 10:12 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Chris Angelico <rosuav@gmail.com>:
>
>> I think this thread is proof that they are to be avoided. The GC
>> wasn't doing its job unless explicitly called on. The true solution is
>> to break the refloop; the quick fix is to call gc.collect(). I stand
>> by the recommendation to put an explanatory comment against the
>> collect call.
>
> What I'm saying is that under most circumstances you shouldn't care if
> the memory consumption goes up and down. The true solution is to not do
> anything about temporary memory consumption. Also, you shouldn't worry
> about breaking circular references. That is also often almost impossible
> to accomplish as so much modern code builds on closures, which generate
> all kinds of circular references under the hood=E2=80=94for your benefit,=
 or
> course.

This isn't a temporary issue, though - see the initial post. After two
hours of five-minutely checks, the computer was wedged. That's a
problem to be solved.

Most of what I do with closures can't create refloops, because the
function isn't referenced from inside itself. You'd need something
like this:

>>> def foo():
    x=3D1
    y=3Dlambda: (x,y)
    return y
>>> len([foo() for _ in range(1000)])
1000
>>> gc.collect()
4000
>>> len([foo() for _ in range(1000)])
1000
>>> gc.collect()
4000
>>> len([foo() for _ in range(1000)])
1000
>>> gc.collect()
4000

That's repeatably creating garbage. But change the function to not
return itself, and there's no loop:

>>> def foo():
    x=3D1
    y=3Dlambda: x
    return y
>>> gc.collect()
0
>>> len([foo() for _ in range(1000)])
1000
>>> gc.collect()
0
>>> len([foo() for _ in range(1000)])
1000
>>> gc.collect()
0

The only even reasonably common case that I can think of is a
recursive nested function:

>>> def foo(x):
    def y(f,x=3Dx):
        f()
        for _ in range(x): y(f,x-1)
    return y

It's a function that returns a function that calls its argument some
number of times, where the number is derived in a stupid way from the
argument to the first function. The whole function is garbage, so it's
not surprising that the GC has to collect it.

>>> len([foo(5) for _ in range(1000)])
1000
>>> gc.collect()
3135
>>> len([foo(5) for _ in range(1000)])
1000
>>> gc.collect()
3135
>>> len([foo(5) for _ in range(1000)])
1000
>>> gc.collect()
3135

Can you give a useful example of a closure that does create a refloop?

ChrisA