Path: csiph.com!usenet.pasdenom.info!goblin1!goblin.stu.neva.ru!uio.no!news.tele.dk!news.tele.dk!small.news.tele.dk!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Peter Otten <__peter__@web.de>
Subject: Re: Generator using item[n-1] + item[n] memory
Date: Sat, 15 Feb 2014 12:27:19 +0100
Organization: None
References: <mailman.6952.1392433921.18130.python-list@python.org> <roy-1037EA.22211114022014@news.panix.com> <CAHkxivccycyvqBsBPJMOTixufiTyv3GNCzU4ELgAUv2F+0EExA@mail.gmail.com> <CALwzidk3RTUqSZ1j3tgRQ7rGOHKsjwUwfE7m696Jp9QDvfhfdA@mail.gmail.com> <CAPTjJmoUA5KC1mOH_NjYQJase2sAyRe9nRF1zT8DbH7pZ=os+g@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7Bit
User-Agent: KNode/4.7.3
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.6983.1392463648.18130.python-list@python.org>
Lines: 74
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:66415

Chris Angelico wrote:

> On Sat, Feb 15, 2014 at 6:27 PM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
>> On Fri, Feb 14, 2014 at 8:31 PM, Nick Timkovich <prometheus235@gmail.com>
>> wrote:
>>> OK, now the trick; adding `data = None` inside the generator works, but
>>> in my actual code I wrap my generator inside of `enumerate()`, which
>>> seems to
>>> obviate the "fix".  Can I get it to play nice or am I forced to count
>>> manually. Is that a feature?
>>
>> Yeah, looks like enumerate also doesn't release its reference to the
>> previous object until after it gets the next one.  You'll just have to
>> make do without.
> 
> You could write your own enumerate function.
> 
> def enumerate(it, i=0):
>     it = iter(it)
>     while True:
>         yield i, next(it)
>         i += 1
> 
> That shouldn't keep any extra references around.

An alternative approach ist to yield weak refs and thus have the generator 
control the object lifetime. This doesn't work with the built-in list type 
though:

import weakref

try:
    from itertools import imap # py2
except ImportError:
    imap = map # py3

N = 0

def log_deleted(*args):
    global N
    N -= 1
    print("deleted, new N: {}".format(N))

def log_created():
    global N
    N += 1
    print("created, new N: {}".format(N))

def weakrefs(f):
    def weakrefs(*args, **kw):
        return imap(lambda x: weakref.proxy(x, log_deleted), f(*args, **kw))
    return weakrefs

class List(list):
    def __str__(self):
        s = str(self[:5])
        if len(self) > 10:
            s = s[:-1] + ", ... ]"
        return s

@weakrefs
def biggen():
    sizes = 1, 1, 10, 1, 1, 10, 10, 1, 1, 10, 10, 20, 1, 1, 20, 20, 1, 1
    for size in sizes:
        data = List([1] * int(size * 1e4))
        log_created()
        yield data
        data = None

if __name__ == "__main__":
    for i, x in enumerate(biggen()):
        print("{} {}".format(i, x))