Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #66353 > unrolled thread

Re: Generator using item[n-1] + item[n] memory

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Generator using item[n-1] + item[n] memory Roy Smith <roy@panix.com> - 2014-02-14 22:21 -0500
    Re: Generator using item[n-1] + item[n] memory Nick Timkovich <prometheus235@gmail.com> - 2014-02-14 21:31 -0600
    Re: Generator using item[n-1] + item[n] memory Ian Kelly <ian.g.kelly@gmail.com> - 2014-02-15 00:27 -0700
    Re: Generator using item[n-1] + item[n] memory Chris Angelico <rosuav@gmail.com> - 2014-02-15 18:41 +1100
    Re: Generator using item[n-1] + item[n] memory Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-02-15 11:03 +0000
    Re: Generator using item[n-1] + item[n] memory Peter Otten <__peter__@web.de> - 2014-02-15 12:27 +0100
    Re: Generator using item[n-1] + item[n] memory Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-02-15 12:28 +0000

#66353 — Re: Generator using item[n-1] + item[n] memory

In article <mailman.6952.1392433921.18130.python-list@python.org>,
 Nick Timkovich <prometheus235@gmail.com> wrote:

> Ah, I think I was equating `yield` too closely with `return` in my head.
>  Whereas `return` results in the destruction of the function's locals,
> `yield` I should have known keeps them around, a la C's `static` functions.
>  Many thanks!

It's not quite like C's static.  With C's static, the static variables 
are per-function.  In Python, yield creates a context per invocation.  
Thus, I can do

def f():
    for i in range(10000):
        yield i

g1 = f()
g2 = f()
print g1.next()
print g1.next()
print g1.next()
print g2.next()
print g1.next()


which prints 0, 1, 2, 0, 3.  There's two contexts active at the same 
time, with a distinct instance of "i" in each one.

[toc] | [next] | [standalone]

#66358

[Multipart message — attachments visible in raw view] — view raw

OK, now the trick; adding `data = None` inside the generator works, but in
my actual code I wrap my generator inside of `enumerate()`, which seems to
obviate the "fix".  Can I get it to play nice or am I forced to count
manually. Is that a feature?


On Fri, Feb 14, 2014 at 9:21 PM, Roy Smith <roy@panix.com> wrote:

> In article <mailman.6952.1392433921.18130.python-list@python.org>,
>  Nick Timkovich <prometheus235@gmail.com> wrote:
>
> > Ah, I think I was equating `yield` too closely with `return` in my head.
> >  Whereas `return` results in the destruction of the function's locals,
> > `yield` I should have known keeps them around, a la C's `static`
> functions.
> >  Many thanks!
>
> It's not quite like C's static.  With C's static, the static variables
> are per-function.  In Python, yield creates a context per invocation.
> Thus, I can do
>
> def f():
>     for i in range(10000):
>         yield i
>
> g1 = f()
> g2 = f()
> print g1.next()
> print g1.next()
> print g1.next()
> print g2.next()
> print g1.next()
>
>
> which prints 0, 1, 2, 0, 3.  There's two contexts active at the same
> time, with a distinct instance of "i" in each one.
> --
> https://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

#66390

On Fri, Feb 14, 2014 at 8:31 PM, Nick Timkovich <prometheus235@gmail.com> wrote:
> OK, now the trick; adding `data = None` inside the generator works, but in
> my actual code I wrap my generator inside of `enumerate()`, which seems to
> obviate the "fix".  Can I get it to play nice or am I forced to count
> manually. Is that a feature?

Yeah, looks like enumerate also doesn't release its reference to the
previous object until after it gets the next one.  You'll just have to
make do without.

[toc] | [prev] | [next] | [standalone]

#66391

On Sat, Feb 15, 2014 at 6:27 PM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
> On Fri, Feb 14, 2014 at 8:31 PM, Nick Timkovich <prometheus235@gmail.com> wrote:
>> OK, now the trick; adding `data = None` inside the generator works, but in
>> my actual code I wrap my generator inside of `enumerate()`, which seems to
>> obviate the "fix".  Can I get it to play nice or am I forced to count
>> manually. Is that a feature?
>
> Yeah, looks like enumerate also doesn't release its reference to the
> previous object until after it gets the next one.  You'll just have to
> make do without.

You could write your own enumerate function.

def enumerate(it, i=0):
    it = iter(it)
    while True:
        yield i, next(it)
        i += 1

That shouldn't keep any extra references around.

ChrisA

[toc] | [prev] | [next] | [standalone]

#66410

On Fri, 14 Feb 2014 22:21:11 -0500, Roy Smith used a generator:

> print g1.next()

Roy, unless you're stuck with Python 2.5 (or older!), you ought to use 
the built-in function next(g1) rather than directly call the next method. 
Not only is this the recommended way to do it, but it's also more future-
proof (Python 3 drops the next method and makes it a dunder method) and 
has more functionality (the next() function takes an optional default 
value).


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#66415

Chris Angelico wrote:

> On Sat, Feb 15, 2014 at 6:27 PM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
>> On Fri, Feb 14, 2014 at 8:31 PM, Nick Timkovich <prometheus235@gmail.com>
>> wrote:
>>> OK, now the trick; adding `data = None` inside the generator works, but
>>> in my actual code I wrap my generator inside of `enumerate()`, which
>>> seems to
>>> obviate the "fix".  Can I get it to play nice or am I forced to count
>>> manually. Is that a feature?
>>
>> Yeah, looks like enumerate also doesn't release its reference to the
>> previous object until after it gets the next one.  You'll just have to
>> make do without.
> 
> You could write your own enumerate function.
> 
> def enumerate(it, i=0):
>     it = iter(it)
>     while True:
>         yield i, next(it)
>         i += 1
> 
> That shouldn't keep any extra references around.

An alternative approach ist to yield weak refs and thus have the generator 
control the object lifetime. This doesn't work with the built-in list type 
though:

import weakref

try:
    from itertools import imap # py2
except ImportError:
    imap = map # py3

N = 0

def log_deleted(*args):
    global N
    N -= 1
    print("deleted, new N: {}".format(N))

def log_created():
    global N
    N += 1
    print("created, new N: {}".format(N))

def weakrefs(f):
    def weakrefs(*args, **kw):
        return imap(lambda x: weakref.proxy(x, log_deleted), f(*args, **kw))
    return weakrefs

class List(list):
    def __str__(self):
        s = str(self[:5])
        if len(self) > 10:
            s = s[:-1] + ", ... ]"
        return s

@weakrefs
def biggen():
    sizes = 1, 1, 10, 1, 1, 10, 10, 1, 1, 10, 10, 20, 1, 1, 20, 20, 1, 1
    for size in sizes:
        data = List([1] * int(size * 1e4))
        log_created()
        yield data
        data = None

if __name__ == "__main__":
    for i, x in enumerate(biggen()):
        print("{} {}".format(i, x))

[toc] | [prev] | [next] | [standalone]

#66428

On 15/02/2014 03:31, Nick Timkovich wrote:
> OK, now the trick; adding `data = None` inside the generator works, but
> in my actual code I wrap my generator inside of `enumerate()`, which
> seems to obviate the "fix".  Can I get it to play nice or am I forced to
> count manually. Is that a feature?
>
>
> On Fri, Feb 14, 2014 at 9:21 PM, Roy Smith <roy@panix.com
> <mailto:roy@panix.com>> wrote:
>
>     In article <mailman.6952.1392433921.18130.python-list@python.org
>     <mailto:mailman.6952.1392433921.18130.python-list@python.org>>,
>       Nick Timkovich <prometheus235@gmail.com
>     <mailto:prometheus235@gmail.com>> wrote:
>
>      > Ah, I think I was equating `yield` too closely with `return` in
>     my head.
>      >  Whereas `return` results in the destruction of the function's
>     locals,
>      > `yield` I should have known keeps them around, a la C's `static`
>     functions.
>      >  Many thanks!
>
>     It's not quite like C's static.  With C's static, the static variables
>     are per-function.  In Python, yield creates a context per invocation.
>     Thus, I can do
>
>     def f():
>          for i in range(10000):
>              yield i
>
>     g1 = f()
>     g2 = f()
>     print g1.next()
>     print g1.next()
>     print g1.next()
>     print g2.next()
>     print g1.next()
>
>
>     which prints 0, 1, 2, 0, 3.  There's two contexts active at the same
>     time, with a distinct instance of "i" in each one.
>     --
>     https://mail.python.org/mailman/listinfo/python-list
>

Nick, please don't top post on this list, thanks.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.

Mark Lawrence

---
This email is free from viruses and malware because avast! Antivirus protection is active.
http://www.avast.com

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web