Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #106219 > unrolled thread

Strange range

Back to article view | Back to comp.lang.python

Contents

  Strange range Marko Rauhamaa <marko@pacujo.net> - 2016-04-01 16:15 +0300
    Re: Strange range Chris Angelico <rosuav@gmail.com> - 2016-04-02 00:24 +1100
    Re: Strange range Steven D'Aprano <steve@pearwood.info> - 2016-04-02 00:26 +1100
      Re: Strange range Marko Rauhamaa <marko@pacujo.net> - 2016-04-01 17:12 +0300
        Re: Strange range Random832 <random832@fastmail.com> - 2016-04-01 10:39 -0400
      Re: Strange range Fabien <fabien.maussion@gmail.com> - 2016-04-01 16:16 +0200
        Re: Strange range Jussi Piitulainen <jussi.piitulainen@helsinki.fi> - 2016-04-01 17:28 +0300
        Re: Strange range Chris Angelico <rosuav@gmail.com> - 2016-04-02 01:31 +1100
          Re: Strange range Marko Rauhamaa <marko@pacujo.net> - 2016-04-01 17:34 +0300
            Re: Strange range Jussi Piitulainen <jussi.piitulainen@helsinki.fi> - 2016-04-01 17:44 +0300
            Re: Strange range Chris Angelico <rosuav@gmail.com> - 2016-04-02 01:45 +1100
              Re: Strange range Marko Rauhamaa <marko@pacujo.net> - 2016-04-01 18:43 +0300
            Re: Strange range Erik <python@lucidity.plus.com> - 2016-04-01 20:58 +0100
              Re: Strange range Marko Rauhamaa <marko@pacujo.net> - 2016-04-01 23:14 +0300
                Re: Strange range Rob Gaddi <rgaddi@highlandtechnology.invalid> - 2016-04-01 20:21 +0000
                  Re: Strange range Marko Rauhamaa <marko@pacujo.net> - 2016-04-01 23:44 +0300
                Re: Strange range Steven D'Aprano <steve@pearwood.info> - 2016-04-02 21:09 +1100
                  Re: Strange range Marko Rauhamaa <marko@pacujo.net> - 2016-04-02 13:48 +0300
            Re: Strange range Ned Batchelder <ned@nedbatchelder.com> - 2016-04-02 12:47 -0700
              Re: Strange range Marko Rauhamaa <marko@pacujo.net> - 2016-04-02 23:44 +0300
                Re: Strange range Chris Angelico <rosuav@gmail.com> - 2016-04-03 07:05 +1000
                  Re: Strange range Marko Rauhamaa <marko@pacujo.net> - 2016-04-03 00:40 +0300
                    Re: Strange range Ned Batchelder <ned@nedbatchelder.com> - 2016-04-02 14:50 -0700
                    Re: Strange range Stephen Hansen <me+python@ixokai.io> - 2016-04-02 23:43 -0700
                      Re: Strange range Marko Rauhamaa <marko@pacujo.net> - 2016-04-03 09:58 +0300
                        Re: Strange range Chris Angelico <rosuav@gmail.com> - 2016-04-03 17:10 +1000
                        Re: Strange range Ethan Furman <ethan@stoneleaf.us> - 2016-04-03 09:28 -0700
                        Re: Strange range Mark Lawrence <breamoreboy@yahoo.co.uk> - 2016-04-04 13:21 +0100
                    Re: Strange range Chris Angelico <rosuav@gmail.com> - 2016-04-03 17:08 +1000
                Re: Strange range Steven D'Aprano <steve@pearwood.info> - 2016-04-03 14:43 +1000
                  Re: Strange range Random832 <random832@fastmail.com> - 2016-04-03 01:20 -0400
                  Re: Strange range Steven D'Aprano <steve@pearwood.info> - 2016-04-03 15:28 +1000
        Re: Strange range Marko Rauhamaa <marko@pacujo.net> - 2016-04-01 17:32 +0300
    Re: Strange range Random832 <random832@fastmail.com> - 2016-04-01 10:42 -0400
    Re: Strange range Chris Angelico <rosuav@gmail.com> - 2016-04-02 01:50 +1100
    Re: Strange range Ian Kelly <ian.g.kelly@gmail.com> - 2016-04-01 08:52 -0600

Page 1 of 2  [1] 2  Next page →

#106219 — Strange range

This seems sane:

    >>> it = iter(range(10))
    >>> for i in it:
    ...   if i >= 3:
    ...     break
    ...
    >>> list(it)
    [4, 5, 6, 7, 8, 9]

As does this:

    >>> it = iter(list(range(10)))
    >>> for i in it:
    ...   if i >= 3:
    ...     break
    ...
    >>> list(it)
    [4, 5, 6, 7, 8, 9]

This not so much:

    >>> it = range(10)
    >>> for i in it:
    ...   if i >= 3:
    ...      break
    ...
    >>> list(it)
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Note to self: range(10) is an iterator factory, not an iterator.


Marko

[toc] | [next] | [standalone]

#106220

On Sat, Apr 2, 2016 at 12:15 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Note to self: range(10) is an iterator factory, not an iterator.

It is an iterable. It is not a factory, as that implies that you call
it. It is an object which, when you ask it for an iterator, gives you
an iterator. That's called an iterable.

ChrisA

[toc] | [prev] | [next] | [standalone]

#106221

On Sat, 2 Apr 2016 12:15 am, Marko Rauhamaa wrote:

> 
> This seems sane:
[...] 
> This not so much:
> 
>     >>> it = range(10)
>     >>> for i in it:
>     ...   if i >= 3:
>     ...      break
>     ...
>     >>> list(it)
>     [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
> 
> Note to self: range(10) is an iterator factory, not an iterator.

Incorrect. range is a lazy sequence.

The only difference between [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] and range(10) is
that the list [0, ..., 9] is calculated eagerly, ahead of time, while
range(10) only generates the values on demand. You can think of range as
equivalent to something close to this:


class Range(object):
    def __init__(self, start, end, step=1):
        self.start = start
        self.end = end
        self.step = step

    def __getitem__(self, index):
        value = self.start + (index-1)*self.step
        if value < self.end:
            return value
        raise IndexError

    def __iter__(self):
        try:
            index = 0
            while True:
                yield self[index]
                index += 1
        except IndexError:
            return



except with more error checking, better bounds checking, support for the
`in` operator, etc.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#106223

Steven D'Aprano <steve@pearwood.info>:

> On Sat, 2 Apr 2016 12:15 am, Marko Rauhamaa wrote:
>> Note to self: range(10) is an iterator factory, not an iterator.
>
> Incorrect. range is a lazy sequence.

Incorrect. You and I agree.

> You can think of range as equivalent to something close to this:
>
> class Range(object):
>     def __init__(self, start, end, step=1):
>         self.start = start
>         self.end = end
>         self.step = step
>
>     def __getitem__(self, index):
>         value = self.start + (index-1)*self.step
>         if value < self.end:
>             return value
>         raise IndexError
>
>     def __iter__(self):
>         try:
>             index = 0
>             while True:
>                 yield self[index]
>                 index += 1
>         except IndexError:
>             return

Yes, I realize it now. I had thought it was:

    def range(start, end=None, step=1):
        if end is None:
            start, end = 0, start
        i = start
        while step * (end - i) > 0:
            yield i
            i += step


Marko

[toc] | [prev] | [next] | [standalone]

#106230

On Fri, Apr 1, 2016, at 10:12, Marko Rauhamaa wrote:
> Yes, I realize it now. I had thought it was:
> 
>     def range(start, end=None, step=1):
>         if end is None:
>             start, end = 0, start
>         i = start
>         while step * (end - i) > 0:
>             yield i
>             i += step

You know the other funny thing? Even people who understand this about
Python 3 ranges, many assume that python 2 xrange was more similar to
your latter example (possibly caused by the fact that python 2 had other
things such as dict.iteritems which _were_ mere iterators)

[toc] | [prev] | [next] | [standalone]

#106224

On 04/01/2016 03:26 PM, Steven D'Aprano wrote:
> Incorrect. range is a lazy sequence.

But how does range "know" that it has to start from scratch again? As in 
this example:

it = range(10)
for i in it:
     if i >= 3:
         break
for i in it:
     # why does it start from zero again?
     print(i)

[toc] | [prev] | [next] | [standalone]

#106225

Fabien writes:

> On 04/01/2016 03:26 PM, Steven D'Aprano wrote:
>> Incorrect. range is a lazy sequence.
>
> But how does range "know" that it has to start from scratch again? As
> in this example:
>
> it = range(10)
> for i in it:
>     if i >= 3:
>         break
> for i in it:
>     # why does it start from zero again?
>     print(i)

The loops are effectively iterating over iter(it) each time. Since "it"
is not already an iterator, iter constructs a new one based on "it".

Try it = iter(range(10)) to see it not start over.

[toc] | [prev] | [next] | [standalone]

#106226

On Sat, Apr 2, 2016 at 1:16 AM, Fabien <fabien.maussion@gmail.com> wrote:
> On 04/01/2016 03:26 PM, Steven D'Aprano wrote:
>>
>> Incorrect. range is a lazy sequence.
>
>
> But how does range "know" that it has to start from scratch again? As in
> this example:
>
> it = range(10)
> for i in it:
>     if i >= 3:
>         break
> for i in it:
>     # why does it start from zero again?
>     print(i)

It's not an iterator. It's an iterable. So every time you call iter()
on it - which is done implicitly by the starting of the 'for' loop -
you get a completely new iterator which will step through the whole
range object.

*A range object is not an iterator.*

ChrisA

[toc] | [prev] | [next] | [standalone]

#106229

Chris Angelico <rosuav@gmail.com>:

> *A range object is not an iterator.*

We now have learned as much.

However, doesn't that extra level of indirection seem like an odd
choice?


Marko

[toc] | [prev] | [next] | [standalone]

#106233

Marko Rauhamaa writes:
> Chris Angelico wrote:
>
>> *A range object is not an iterator.*
>
> We now have learned as much.
>
> However, doesn't that extra level of indirection seem like an odd
> choice?

I think it's brilliant. The range object behaves like a tuple containing
the numbers in the range, one can index it, slice it, zip it, for it,
like it, without it taking up the space or getting consumed in the
process.

[toc] | [prev] | [next] | [standalone]

#106234

On Sat, Apr 2, 2016 at 1:34 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Chris Angelico <rosuav@gmail.com>:
>
>> *A range object is not an iterator.*
>
> We now have learned as much.
>
> However, doesn't that extra level of indirection seem like an odd
> choice?

No; a range object is an entity in itself. You can test if something's
within the range:

>>> 5 in range(2,10)
True
>>> 5 in range(2,10,2)
False

You can ask how many numbers are in the range:

>>> len(range(2,10,2))
4

You can even ask what position a number would be in, if you index
through the range:

>>> range(3,100,3).index(57)
18

Iterators can't do any of this, except the 'in' check, which is
destructive and O(N).

ChrisA

[toc] | [prev] | [next] | [standalone]

#106239

Chris Angelico <rosuav@gmail.com>:

> On Sat, Apr 2, 2016 at 1:34 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
>> However, doesn't that extra level of indirection seem like an odd
>> choice?
>
> No; a range object is an entity in itself. You can test if something's
> within the range:
>
>>>> 5 in range(2,10)
> True
>>>> 5 in range(2,10,2)
> False
>
> You can ask how many numbers are in the range:
>
>>>> len(range(2,10,2))
> 4
>
> You can even ask what position a number would be in, if you index
> through the range:
>
>>>> range(3,100,3).index(57)
> 18
>
> Iterators can't do any of this, except the 'in' check, which is
> destructive and O(N).

I can't think of a situation where I would have needed those things. I'm
hard-pressed to find an example of noniterator usage of range in the
standard library, for example. Here the only counterexample that caught
my eye among the first couple of hundred of occurrences:

   reversed(range(len(self)))


Marko

[toc] | [prev] | [next] | [standalone]

#106251

On 01/04/16 15:34, Marko Rauhamaa wrote:
> Chris Angelico <rosuav@gmail.com>:
>
>> *A range object is not an iterator.*
>
> We now have learned as much.
>
> However, doesn't that extra level of indirection seem like an odd
> choice?

If you write your own class which has an __iter__ method, would you expect:

   >>> o = MyClass()
   >>> i1 = iter(o)
   >>> i2 = iter(o)
   >>> list(i1)
   ['foo', 'bar', 'baz']
   >>> list(i2)
   []

?

Or, would you expect both iterators to be independent and both return 
the 'foo', 'bar', 'baz' sequence (where that is the hypothetical result 
of iterating over your object)?

If you now replace MyClass() with range(10), why would you expect the 
two iterators to be related?

E.

[toc] | [prev] | [next] | [standalone]

#106252

Erik <python@lucidity.plus.com>:

> On 01/04/16 15:34, Marko Rauhamaa wrote:
>> Chris Angelico <rosuav@gmail.com>:
>>
>>> *A range object is not an iterator.*
>>
>> We now have learned as much.
>>
>> However, doesn't that extra level of indirection seem like an odd
>> choice?
>
> [...]
>
> If you write your own class which has an __iter__ method, would you
> [...] expect both iterators to be independent and both return the
> 'foo', 'bar', 'baz' sequence (where that is the hypothetical result of
> iterating over your object)?
>
> If you now replace MyClass() with range(10), why would you expect the
> two iterators to be related?

I simply had thought of range() returning an iterator. I would expect an
iterator to behave like one.

There's a bit of a cognitive dissonance between iterables and iterators.
On the one hand, they behave identically in many contexts. On the other
hand, the distinction is crucial in some special cases.

(Somehow, the difference between iterables and iterators is analogous
with the difference between C's arrays and pointers.)


Marko

[toc] | [prev] | [next] | [standalone]

#106253

Marko Rauhamaa wrote:

> Erik <python@lucidity.plus.com>:
>
>> On 01/04/16 15:34, Marko Rauhamaa wrote:
>>> Chris Angelico <rosuav@gmail.com>:
>>>
>>>> *A range object is not an iterator.*
>>>
>>> We now have learned as much.
>>>
>>> However, doesn't that extra level of indirection seem like an odd
>>> choice?
>>
>> [...]
>>
>> If you write your own class which has an __iter__ method, would you
>> [...] expect both iterators to be independent and both return the
>> 'foo', 'bar', 'baz' sequence (where that is the hypothetical result of
>> iterating over your object)?
>>
>> If you now replace MyClass() with range(10), why would you expect the
>> two iterators to be related?
>
> I simply had thought of range() returning an iterator. I would expect an
> iterator to behave like one.
>
> There's a bit of a cognitive dissonance between iterables and iterators.
> On the one hand, they behave identically in many contexts. On the other
> hand, the distinction is crucial in some special cases.

You're missing a key point.  All (well-behaved) iterators are iterables,
with their __iter__ method returning themselves.

for x in y:
  ...

implies:

try:
  _it = iter(y)
  while True:
    x = next(_it)
    ...
except StopIteration:
  pass

That's true for any iterable y, including a y which is itself an
iterator.  You still call iter() on it, and get _it, the iterator over
y, which if y is an iterator is the same thing as y.

-- 
Rob Gaddi, Highland Technology -- www.highlandtechnology.com

Email address domain is currently out of order.  See above to fix.

[toc] | [prev] | [next] | [standalone]

#106260

Rob Gaddi <rgaddi@highlandtechnology.invalid>:

> Marko Rauhamaa wrote:
>> There's a bit of a cognitive dissonance between iterables and iterators.
>> On the one hand, they behave identically in many contexts. On the other
>> hand, the distinction is crucial in some special cases.
>
> You're missing a key point.  All (well-behaved) iterators are iterables,
> with their __iter__ method returning themselves.

I don't know what I'm missing.


Marko

[toc] | [prev] | [next] | [standalone]

#106282

On Sat, 2 Apr 2016 07:14 am, Marko Rauhamaa wrote:

> There's a bit of a cognitive dissonance between iterables and iterators.
> On the one hand, they behave identically in many contexts. On the other
> hand, the distinction is crucial in some special cases.

Iterable just means "something which can be iterated over". Anything which
you can feed directly into a for-loop:

for x in obj: ...

is by definition an iterable. The particular mechanism by which iteration is
performed is unspecified.

In Python today, there are two specific mechanisms, although in practice
only one commonly is used. The older of the two is the "Sequence Protocol",
which says that iteration is performed by repeatedly fetching an item from
the object with an incrementing index until IndexError is raised:

x = obj[0]
x = obj[1]
x = obj[2]
...


The newer, and more common, mechanism is to call iter(obj) to get an
iterator, then repeatedly call next(iterator) until it raises
StopIteration.

it = iter(obj)
x = next(it)
x = next(it)
x = next(it)
...


So what's an iterator? Any object which obeys the "Iterator Protocol". The
rules for the protocol are:

- the object must provide two methods, __iter__ and __next__;

- __iter__ must return self;

- __next__ must return the next value in the sequence, 
  or raise StopIteration.


range objects themselves do not obey the iterator protocol, which makes them
not iterators:

py> o = range(23)
py> iter(o) is o
False
py> hasattr(o, '__next__')
False


But they are *iterable* since you can call iter on a range object and get an
iterator:

py> it = iter(o)
py> iter(it) is it
True
py> hasattr(it, '__next__')
True



> (Somehow, the difference between iterables and iterators is analogous
> with the difference between C's arrays and pointers.)

I don't understand this analogy. Can you explain please?


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#106284

Steven D'Aprano <steve@pearwood.info>:

> On Sat, 2 Apr 2016 07:14 am, Marko Rauhamaa wrote:
>> (Somehow, the difference between iterables and iterators is analogous
>> with the difference between C's arrays and pointers.)
>
> I don't understand this analogy. Can you explain please?

In numerous contexts,

   T a[N]

and

   T *a

are interchangeable. In fact, C has no rvalue notation for an array. For
any other type, this works:

   T a, b;
   a = b;

However, if T is an array type, the compiler complains:

   error: assignment to expression with array type

This C innovation of blurring the lines between arrays and pointers is
very different of the type system of Pascal, whose arrays behave like
any other type.

C could have treated arrays like other types without any loss of
generality. Then you'd have to write:

   T a[N], *b;
   b = &a[0];

for:

   T a[N], *b;
   b = a;

Semantically, as well, C arrays are iterables, and pointers are used to
iterate over the elements of an array.

Similarly, Python could have kept iterables and iterators in their
separate corners by specifying:

 * iter(iterable) returns an iterator

 * iter(iterator) typically raises an Exception

 * the for statement requires an iterator

Then, you'd have to write:

   for i in iter([1, 2, 3]):
       ...


Not recommending anything one way or another.


Marko

[toc] | [prev] | [next] | [standalone]

#106308

On Friday, April 1, 2016 at 10:34:50 AM UTC-4, Marko Rauhamaa wrote:
> Chris Angelico <rosuav@gmail.com>:
> 
> > *A range object is not an iterator.*
> 
> We now have learned as much.
> 
> However, doesn't that extra level of indirection seem like an odd
> choice?

I agree that it is surprising (and confusing) at first.  It took me
some getting used to.

Now I think of it like this: An iterable is a thing that could be
iterated, that is, it has a sequence of things.  An iterator holds the
current status of a thing that is being iterated.  For example, the
pages of a book are an iterable (we could iterate over them), and a
bookmark is an iterator for the pages of a book: it knows where we are
in the iteration.

This analogy illuminates an important point: a single iterable can have
a number of active iterators working over it at once, just as a book can
have a number of bookmarks in it at once.

    nums = [1, 2, 3]
    for i in nums:
        for j in nums:
            print i, j

This prints all the pairs of numbers, because the iterator in the first
loop is independent of the iterator(s) in the second loop, even though
they are iterating over the same iterator (the nums list).  Without the
extra indirection of iterators over iterables, this code would get
tangled up.

--Ned.

[toc] | [prev] | [next] | [standalone]

#106310

Ned Batchelder <ned@nedbatchelder.com>:

> This analogy illuminates an important point: a single iterable can have
> a number of active iterators working over it at once, just as a book can
> have a number of bookmarks in it at once.
>
>     nums = [1, 2, 3]
>     for i in nums:
>         for j in nums:
>             print i, j
>
> This prints all the pairs of numbers, because the iterator in the first
> loop is independent of the iterator(s) in the second loop, even though
> they are iterating over the same iterator (the nums list).  Without the
> extra indirection of iterators over iterables, this code would get
> tangled up.

I don't have a problem with a list being a "reiterable." I only was
surprised about range(), which I had thought to be a plain,
down-to-earth iterator. There's barely any other practical use for a
range, I believe.


Marko

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python

csiph-web