Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #77831 > unrolled thread

Iterator, modify data in loop body

Back to article view | Back to comp.lang.python

Contents

  Iterator, modify data in loop body Michael Welle <mwe012008@gmx.net> - 2014-09-13 08:09 +0200
    Re: Iterator, modify data in loop body Chris Angelico <rosuav@gmail.com> - 2014-09-13 16:22 +1000
      Re: Iterator, modify data in loop body Michael Welle <mwe012008@gmx.net> - 2014-09-13 09:01 +0200
        Re: Iterator, modify data in loop body Chris Angelico <rosuav@gmail.com> - 2014-09-13 17:22 +1000
          Re: Iterator, modify data in loop body Michael Welle <mwe012008@gmx.net> - 2014-09-13 09:39 +0200
            Re: Iterator, modify data in loop body Ian Kelly <ian.g.kelly@gmail.com> - 2014-09-13 09:27 -0600
              Re: Iterator, modify data in loop body Michael Welle <mwe012008@gmx.net> - 2014-09-13 18:58 +0200
            Re: Iterator, modify data in loop body Chris Angelico <rosuav@gmail.com> - 2014-09-14 01:32 +1000
          Re: Iterator, modify data in loop body Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2014-09-16 10:49 +0200
            Re: Iterator, modify data in loop body Chris Angelico <rosuav@gmail.com> - 2014-09-17 01:19 +1000
    Re: Iterator, modify data in loop body Peter Otten <__peter__@web.de> - 2014-09-13 09:34 +0200
      Re: Iterator, modify data in loop body Michael Welle <mwe012008@gmx.net> - 2014-09-13 09:55 +0200

#77831 — Iterator, modify data in loop body

Hello,

I want to create an iterator it=iter(list) and control a for-loop with
it. Is it save to append elements to the list in the body of the
for-loop or is the behaviour undefined then? PEP234 notes that once the
iterator has signaled exhaustion, subsequent calls of next() should not
change that state. That suggests that it is possible to modify the list
during the iterator's livetime.

Ex.:

foo = [1,2,3,4]
it = iter(foo)

for e in it:
    if e % 2 == 0:
        x.append(e)


Regards
hmw

-- 
biff4emacsen - A biff-like tool for (X)Emacs
http://www.c0t0d0s0.de/biff4emacsen/biff4emacsen.html
Flood - Your friendly network packet generator
http://www.c0t0d0s0.de/flood/flood.html

[toc] | [next] | [standalone]

#77832

On Sat, Sep 13, 2014 at 4:09 PM, Michael Welle <mwe012008@gmx.net> wrote:
> foo = [1,2,3,4]
> it = iter(foo)
>
> for e in it:
>     if e % 2 == 0:
>         x.append(e)

A better way to do this is with a list comprehension:

x = [e for e in foo if e %2 == 0]

Modifying something that you're iterating over is unspecified, I
believe. Certainly it's not a good idea.

ChrisA

[toc] | [prev] | [next] | [standalone]

#77835

Hello,

Chris Angelico <rosuav@gmail.com> writes:

> On Sat, Sep 13, 2014 at 4:09 PM, Michael Welle <mwe012008@gmx.net> wrote:
>> foo = [1,2,3,4]
>> it = iter(foo)
>>
>> for e in it:
>>     if e % 2 == 0:
>>         x.append(e)
>
> A better way to do this is with a list comprehension:
>
> x = [e for e in foo if e %2 == 0]
ideed, this works for the minimal example. In a real application list
comprehension might be a bit unhandy, because there is a lot of code
involved. 


> Modifying something that you're iterating over is unspecified, I
> believe. 
Hm, if only I could find that in the documentation :). I've found some
cases in the documentation where it is made clear that modifying the
data object is 'verboten' and some cases bail out with an exception.


> Certainly it's not a good idea.
It depends on the use case I think. For some algorithms it feels natural
to just append at the end of the list while consuming elements from the
front. I think a deque supports that as well.

Regards
hmw

-- 
biff4emacsen - A biff-like tool for (X)Emacs
http://www.c0t0d0s0.de/biff4emacsen/biff4emacsen.html
Flood - Your friendly network packet generator
http://www.c0t0d0s0.de/flood/flood.html

[toc] | [prev] | [next] | [standalone]

#77836

On Sat, Sep 13, 2014 at 5:01 PM, Michael Welle <mwe012008@gmx.net> wrote:
> ideed, this works for the minimal example. In a real application list
> comprehension might be a bit unhandy, because there is a lot of code
> involved.

Sure. Sometimes, cutting something down for posting makes a completely
different solution possible, and that doesn't much help. :)

> It depends on the use case I think. For some algorithms it feels natural
> to just append at the end of the list while consuming elements from the
> front. I think a deque supports that as well.

In that case, don't iterate over the list at all. Do something like this:

while lst:
    element = lst.pop(0)
    # work with element
    lst.append(new_element)

There's no mutation-while-iterating here, and it's clear that you'll
keep going until there's absolutely nothing left.

ChrisA

[toc] | [prev] | [next] | [standalone]

#77838

Hello,

Chris Angelico <rosuav@gmail.com> writes:

> On Sat, Sep 13, 2014 at 5:01 PM, Michael Welle <mwe012008@gmx.net> wrote:
>> ideed, this works for the minimal example. In a real application list
>> comprehension might be a bit unhandy, because there is a lot of code
>> involved.
>
> Sure. Sometimes, cutting something down for posting makes a completely
> different solution possible, and that doesn't much help. :)
and sometimes new and interesting ideas come up doing so ;).


>> It depends on the use case I think. For some algorithms it feels natural
>> to just append at the end of the list while consuming elements from the
>> front. I think a deque supports that as well.
>
> In that case, don't iterate over the list at all. Do something like this:
>
> while lst:
>     element = lst.pop(0)
>     # work with element
>     lst.append(new_element)
>
> There's no mutation-while-iterating here, and it's clear that you'll
> keep going until there's absolutely nothing left.
Ah, that looks like a good approach, thank you.

Regards
hmw

-- 
biff4emacsen - A biff-like tool for (X)Emacs
http://www.c0t0d0s0.de/biff4emacsen/biff4emacsen.html
Flood - Your friendly network packet generator
http://www.c0t0d0s0.de/flood/flood.html

[toc] | [prev] | [next] | [standalone]

#77848

On Sat, Sep 13, 2014 at 1:39 AM, Michael Welle <mwe012008@gmx.net> wrote:
>> In that case, don't iterate over the list at all. Do something like this:
>>
>> while lst:
>>     element = lst.pop(0)
>>     # work with element
>>     lst.append(new_element)
>>
>> There's no mutation-while-iterating here, and it's clear that you'll
>> keep going until there's absolutely nothing left.
> Ah, that looks like a good approach, thank you.

Also note that this approach (appending to the end and popping from
the front) will be more efficient using a collections.deque than a
list.

[toc] | [prev] | [next] | [standalone]

#77850

Hello,

Ian Kelly <ian.g.kelly@gmail.com> writes:

> On Sat, Sep 13, 2014 at 1:39 AM, Michael Welle <mwe012008@gmx.net> wrote:
>>> In that case, don't iterate over the list at all. Do something like this:
>>>
>>> while lst:
>>>     element = lst.pop(0)
>>>     # work with element
>>>     lst.append(new_element)
>>>
>>> There's no mutation-while-iterating here, and it's clear that you'll
>>> keep going until there's absolutely nothing left.
>> Ah, that looks like a good approach, thank you.
>
> Also note that this approach (appending to the end and popping from
> the front) will be more efficient using a collections.deque than a
> list.
sure, the references hold by the list will be copied from position n to
position n-1 after popping the 0th element.

Regards
hmw

-- 
biff4emacsen - A biff-like tool for (X)Emacs
http://www.c0t0d0s0.de/biff4emacsen/biff4emacsen.html
Flood - Your friendly network packet generator
http://www.c0t0d0s0.de/flood/flood.html

[toc] | [prev] | [next] | [standalone]

#77849

On Sun, Sep 14, 2014 at 1:27 AM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
> On Sat, Sep 13, 2014 at 1:39 AM, Michael Welle <mwe012008@gmx.net> wrote:
>>> In that case, don't iterate over the list at all. Do something like this:
>>>
>>> while lst:
>>>     element = lst.pop(0)
>>>     # work with element
>>>     lst.append(new_element)
>>>
>>> There's no mutation-while-iterating here, and it's clear that you'll
>>> keep going until there's absolutely nothing left.
>> Ah, that looks like a good approach, thank you.
>
> Also note that this approach (appending to the end and popping from
> the front) will be more efficient using a collections.deque than a
> list.

Sure it will - that's kinda the point of a double-ended queue, to
avoid all the inefficient movements :) But the concept is still the
same: do repeated mutations rather than iteration. Either that, or
iterate and build up a new list, either with a comprehension or with
something like this:

newlst = []
for element in lst:
    # work with element
    newlst.append(new_element)

ChrisA

[toc] | [prev] | [next] | [standalone]

#77935

Am 13.09.2014 09:22 schrieb Chris Angelico:

> In that case, don't iterate over the list at all. Do something like this:
>
> while lst:
>      element = lst.pop(0)
>      # work with element
>      lst.append(new_element)

And if you don't like that, define a

def iter_pop(lst):
     while lst:
         yield lst.pop(0)

and you can do

     for element in iter_pop(lst):
         # work with element
         lst.append(new_element)


Thomas

[toc] | [prev] | [next] | [standalone]

#77937

On Tue, Sep 16, 2014 at 6:49 PM, Thomas Rachel
<nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de>
wrote:
> Am 13.09.2014 09:22 schrieb Chris Angelico:
>
>> In that case, don't iterate over the list at all. Do something like this:
>>
>> while lst:
>>      element = lst.pop(0)
>>      # work with element
>>      lst.append(new_element)
>
>
> And if you don't like that, define a
>
> def iter_pop(lst):
>     while lst:
>         yield lst.pop(0)
>
> and you can do
>
>     for element in iter_pop(lst):

But that's exactly the same thing, with another level of indirection.
It certainly isn't the advantage you'd expect from an iterator, namely
that it simply stores a marker that gets advanced to the next element.
Popping the 0th element is costly, wrapping it into an iterator
conceals that.

ChrisA

[toc] | [prev] | [next] | [standalone]

#77837

Michael Welle wrote:

> I want to create an iterator it=iter(list) and control a for-loop with
> it. Is it save to append elements to the list in the body of the
> for-loop or is the behaviour undefined then? PEP234 notes that once the
> iterator has signaled exhaustion, subsequent calls of next() should not
> change that state. That suggests that it is possible to modify the list
> during the iterator's livetime.

It's possible, but usually not a good idea. Especially inserting or deleting 
elements before the current position of the iterator (you can think of it as 
an index into the list) gives results that are usually unexpected:

>>> items = [1, "remove me", 2, "remove me", "remove me", 3, 4]
>>> for item in items:
...     if item == "remove me":
...         items.remove(item)
... 
>>> items
[1, 2, 'remove me', 3, 4]

Pro tip: don't do it even when it's possible.

> Ex.:
> 
> foo = [1,2,3,4]
> it = iter(foo)
>
> for e in it:
>     if e % 2 == 0:
>         x.append(e)

I don't see how the example is related to the question. Did you mean

   foo.append(e)

? With that modification the loop would run "forever" because you keep 
appending items that satisfy the condition e % 2 == 0.

> it = iter(foo)

Normally you would just iterate over foo; the only use-case where you'd 
create an iterator explicitly is when you want to skip some items:

it = iter(items)
for e in it:
    if skip_next(e):
        next(it)

[toc] | [prev] | [next] | [standalone]

#77840

Hello,

Peter Otten <__peter__@web.de> writes:

> Michael Welle wrote:
>
>> I want to create an iterator it=iter(list) and control a for-loop with
>> it. Is it save to append elements to the list in the body of the
>> for-loop or is the behaviour undefined then? PEP234 notes that once the
>> iterator has signaled exhaustion, subsequent calls of next() should not
>> change that state. That suggests that it is possible to modify the list
>> during the iterator's livetime.
>
> It's possible, but usually not a good idea. Especially inserting or deleting 
> elements before the current position of the iterator (you can think of it as 
> an index into the list) gives results that are usually unexpected:
jepp, that is clear.


[...]
>> Ex.:
>> 
>> foo = [1,2,3,4]
>> it = iter(foo)
>>
>> for e in it:
>>     if e % 2 == 0:
>>         x.append(e)
>
> I don't see how the example is related to the question. Did you mean
>
>    foo.append(e)
Sorry, you are right.


> ? With that modification the loop would run "forever" because you keep 
> appending items that satisfy the condition e % 2 == 0.
You are right again. It's intended, because it's one of the simplest
examples to illustrate the question.

Regards
hmw

-- 
biff4emacsen - A biff-like tool for (X)Emacs
http://www.c0t0d0s0.de/biff4emacsen/biff4emacsen.html
Flood - Your friendly network packet generator
http://www.c0t0d0s0.de/flood/flood.html

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web