Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #66415

Re: Generator using item[n-1] + item[n] memory

Show key headers only | View raw

Chris Angelico wrote:

> On Sat, Feb 15, 2014 at 6:27 PM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
>> On Fri, Feb 14, 2014 at 8:31 PM, Nick Timkovich <prometheus235@gmail.com>
>> wrote:
>>> OK, now the trick; adding `data = None` inside the generator works, but
>>> in my actual code I wrap my generator inside of `enumerate()`, which
>>> seems to
>>> obviate the "fix".  Can I get it to play nice or am I forced to count
>>> manually. Is that a feature?
>>
>> Yeah, looks like enumerate also doesn't release its reference to the
>> previous object until after it gets the next one.  You'll just have to
>> make do without.
> 
> You could write your own enumerate function.
> 
> def enumerate(it, i=0):
>     it = iter(it)
>     while True:
>         yield i, next(it)
>         i += 1
> 
> That shouldn't keep any extra references around.

An alternative approach ist to yield weak refs and thus have the generator 
control the object lifetime. This doesn't work with the built-in list type 
though:

import weakref

try:
    from itertools import imap # py2
except ImportError:
    imap = map # py3

N = 0

def log_deleted(*args):
    global N
    N -= 1
    print("deleted, new N: {}".format(N))

def log_created():
    global N
    N += 1
    print("created, new N: {}".format(N))

def weakrefs(f):
    def weakrefs(*args, **kw):
        return imap(lambda x: weakref.proxy(x, log_deleted), f(*args, **kw))
    return weakrefs

class List(list):
    def __str__(self):
        s = str(self[:5])
        if len(self) > 10:
            s = s[:-1] + ", ... ]"
        return s

@weakrefs
def biggen():
    sizes = 1, 1, 10, 1, 1, 10, 10, 1, 1, 10, 10, 20, 1, 1, 20, 20, 1, 1
    for size in sizes:
        data = List([1] * int(size * 1e4))
        log_created()
        yield data
        data = None

if __name__ == "__main__":
    for i, x in enumerate(biggen()):
        print("{} {}".format(i, x))

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Re: Generator using item[n-1] + item[n] memory Roy Smith <roy@panix.com> - 2014-02-14 22:21 -0500
  Re: Generator using item[n-1] + item[n] memory Nick Timkovich <prometheus235@gmail.com> - 2014-02-14 21:31 -0600
  Re: Generator using item[n-1] + item[n] memory Ian Kelly <ian.g.kelly@gmail.com> - 2014-02-15 00:27 -0700
  Re: Generator using item[n-1] + item[n] memory Chris Angelico <rosuav@gmail.com> - 2014-02-15 18:41 +1100
  Re: Generator using item[n-1] + item[n] memory Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-02-15 11:03 +0000
  Re: Generator using item[n-1] + item[n] memory Peter Otten <__peter__@web.de> - 2014-02-15 12:27 +0100
  Re: Generator using item[n-1] + item[n] memory Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-02-15 12:28 +0000

csiph-web