Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #42388 > unrolled thread

collections.Iterator __subclasshook__ does not check if next() is callable

Back to article view | Back to comp.lang.python

Contents

  collections.Iterator __subclasshook__ does not check if next() is callable Byron Ruth <bjruth@gmail.com> - 2013-03-31 10:57 -0700
    Re: collections.Iterator __subclasshook__ does not check if next() is callable Byron Ruth <bjruth@gmail.com> - 2013-03-31 12:44 -0700
    Re: collections.Iterator __subclasshook__ does not check if next() is callable Terry Jan Reedy <tjreedy@udel.edu> - 2013-03-31 15:47 -0400
      Re: collections.Iterator __subclasshook__ does not check if next() is callable Byron Ruth <bjruth@gmail.com> - 2013-03-31 12:58 -0700
      Re: collections.Iterator __subclasshook__ does not check if next() is callable Byron Ruth <bjruth@gmail.com> - 2013-03-31 12:58 -0700
    Re: collections.Iterator __subclasshook__ does not check if next() is callable Ethan Furman <ethan@stoneleaf.us> - 2013-03-31 17:02 -0700

#42388 — collections.Iterator __subclasshook__ does not check if next() is callable

I submitted this as bug last night: http://bugs.python.org/issue17584 and was *honored* to be rejected by Raymond Hettinger. However, I would like feedback on whether my concern (this bug) is justified and clarity if not.

Consider:

```python
class A(object):
    def __init__(self):
        self.r = iter(range(5))
    def __iter__(self):
        return self
    @property
    def next(self):
        return next(self.r)
```

The `next` method is a property, however:

```python
from collections import Iterator
a = A()
isinstance(a, Iterator) # True
next(a) # TypeError: 'int' object is not callable
```

I am using `collections.Iterator` as the means to check if the object is an iterator, however I am not sure if that is _root_ problem here. My understanding of the iterator protocol is that is assumes the __iter__ and next *methods* are implemented. In the example, `A.next` is defined as a property, but is still identified as an iterator. To me, this is incorrect behavior since it's not conforming to the iterator protocol requirements (i.e. a `next` method, not a property).

Raymond stated: "The design of ABCs are to check for the existence to required named; none of them verify the signature." I think I understand _why_ this is the case.. but I downstream libraries use `collections.Iterator` to determine if an object _is one_: see https://github.com/django/django/blob/master/django/utils/itercompat.py#L22-L31

Who's job is it to check if `next` (and technically `__iter__`) are methods?

[toc] | [next] | [standalone]

#42396

Raymond's replied to my follow-up and made me realize that the `next` property could return a callable and it would be transparent to the caller.

On Sunday, March 31, 2013 1:57:08 PM UTC-4, Byron Ruth wrote:
> I submitted this as bug last night: http://bugs.python.org/issue17584 and was *honored* to be rejected by Raymond Hettinger. However, I would like feedback on whether my concern (this bug) is justified and clarity if not.
> 
> 
> 
> Consider:
> 
> 
> 
> ```python
> 
> class A(object):
> 
>     def __init__(self):
> 
>         self.r = iter(range(5))
> 
>     def __iter__(self):
> 
>         return self
> 
>     @property
> 
>     def next(self):
> 
>         return next(self.r)
> 
> ```
> 
> 
> 
> The `next` method is a property, however:
> 
> 
> 
> ```python
> 
> from collections import Iterator
> 
> a = A()
> 
> isinstance(a, Iterator) # True
> 
> next(a) # TypeError: 'int' object is not callable
> 
> ```
> 
> 
> 
> I am using `collections.Iterator` as the means to check if the object is an iterator, however I am not sure if that is _root_ problem here. My understanding of the iterator protocol is that is assumes the __iter__ and next *methods* are implemented. In the example, `A.next` is defined as a property, but is still identified as an iterator. To me, this is incorrect behavior since it's not conforming to the iterator protocol requirements (i.e. a `next` method, not a property).
> 
> 
> 
> Raymond stated: "The design of ABCs are to check for the existence to required named; none of them verify the signature." I think I understand _why_ this is the case.. but I downstream libraries use `collections.Iterator` to determine if an object _is one_: see https://github.com/django/django/blob/master/django/utils/itercompat.py#L22-L31
> 
> 
> 
> Who's job is it to check if `next` (and technically `__iter__`) are methods?

[toc] | [prev] | [next] | [standalone]

#42398

On 3/31/2013 1:57 PM, Byron Ruth wrote:
> I submitted this as bug last night: http://bugs.python.org/issue17584 and was *honored* to be rejected by Raymond Hettinger. However, I would like feedback on whether my concern (this bug) is justified and clarity if not.
>
> Consider:
>
> ```python
> class A(object):
>      def __init__(self):
>          self.r = iter(range(5))
>      def __iter__(self):
>          return self
>      @property
>      def next(self):
>          return next(self.r)
> ```
>
> The `next` method is a property, however:

A competent Python programmer should not do that. In Py3, the method is 
properly renamed '__next__', which should make doing that accidentally 
even less likely.

>
> ```python
> from collections import Iterator
> a = A()
> isinstance(a, Iterator) # True
> next(a) # TypeError: 'int' object is not callable
> ```
>
> I am using `collections.Iterator` as the means to check if the object is an iterator,

Being an Iterator only means that it *might* be an iterator.

 > however I am not sure if that is _root_ problem here. My 
understanding of the iterator protocol is that is assumes the __iter__ 
and next *methods* are implemented. In the example, `A.next` is defined 
as a property, but is still identified as an iterator. To me, this is 
incorrect behavior since it's not conforming to the iterator protocol 
requirements (i.e. a `next` method, not a property).

There is more to any protocol than can be statically checked.

> Raymond stated: "The design of ABCs are to check for the existence to required named; none of them verify the signature."

Having the required attributes is currently the definition of being an 
instance of an ABC. Adding 'not a property' would be possible. but 
hardly worthwhile. Checking signatures would be worthwhile, but 
signatures are not yet available to Python for C-coded methods, let 
alone other implementations.

  I think I understand _why_ this is the case.. but I downstream 
libraries use `collections.Iterator` to determine if an object _is one_: 
see 
https://github.com/django/django/blob/master/django/utils/itercompat.py#L22-L31
>
> Who's job is it to check if `next` (and technically `__iter__`) are methods?

The programmer, and a user who does not trust the competence of the 
programmer. But this is the least of the possible errors.

--
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#42399

Thanks for responding Terry.

I can assure you I did not initially realize both the `next` and the `__iter__` methods were implemented when I ran into my original problem. I saw a behavior and had to work backwards to realize why it was behaving the way it was (the comparison against Iterator). Once I realized this, the behavior made complete sense. It just dawned on me the fact that `next` was not being checked to be callable (I was surprised by this at the time) which is why I investigated the `Iterator.__subclasshook__` and assumed it was behaving incorrectly based on my assumptions.

On Sunday, March 31, 2013 3:47:07 PM UTC-4, Terry Jan Reedy wrote:
> On 3/31/2013 1:57 PM, Byron Ruth wrote:
> 
> > I submitted this as bug last night: http://bugs.python.org/issue17584 and was *honored* to be rejected by Raymond Hettinger. However, I would like feedback on whether my concern (this bug) is justified and clarity if not.
> 
> >
> 
> > Consider:
> 
> >
> 
> > ```python
> 
> > class A(object):
> 
> >      def __init__(self):
> 
> >          self.r = iter(range(5))
> 
> >      def __iter__(self):
> 
> >          return self
> 
> >      @property
> 
> >      def next(self):
> 
> >          return next(self.r)
> 
> > ```
> 
> >
> 
> > The `next` method is a property, however:
> 
> 
> 
> A competent Python programmer should not do that. In Py3, the method is 
> 
> properly renamed '__next__', which should make doing that accidentally 
> 
> even less likely.
> 
> 
> 
> >
> 
> > ```python
> 
> > from collections import Iterator
> 
> > a = A()
> 
> > isinstance(a, Iterator) # True
> 
> > next(a) # TypeError: 'int' object is not callable
> 
> > ```
> 
> >
> 
> > I am using `collections.Iterator` as the means to check if the object is an iterator,
> 
> 
> 
> Being an Iterator only means that it *might* be an iterator.
> 
> 
> 
>  > however I am not sure if that is _root_ problem here. My 
> 
> understanding of the iterator protocol is that is assumes the __iter__ 
> 
> and next *methods* are implemented. In the example, `A.next` is defined 
> 
> as a property, but is still identified as an iterator. To me, this is 
> 
> incorrect behavior since it's not conforming to the iterator protocol 
> 
> requirements (i.e. a `next` method, not a property).
> 
> 
> 
> There is more to any protocol than can be statically checked.
> 
> 
> 
> > Raymond stated: "The design of ABCs are to check for the existence to required named; none of them verify the signature."
> 
> 
> 
> Having the required attributes is currently the definition of being an 
> 
> instance of an ABC. Adding 'not a property' would be possible. but 
> 
> hardly worthwhile. Checking signatures would be worthwhile, but 
> 
> signatures are not yet available to Python for C-coded methods, let 
> 
> alone other implementations.
> 
> 
> 
>   I think I understand _why_ this is the case.. but I downstream 
> 
> libraries use `collections.Iterator` to determine if an object _is one_: 
> 
> see 
> 
> https://github.com/django/django/blob/master/django/utils/itercompat.py#L22-L31
> 
> >
> 
> > Who's job is it to check if `next` (and technically `__iter__`) are methods?
> 
> 
> 
> The programmer, and a user who does not trust the competence of the 
> 
> programmer. But this is the least of the possible errors.
> 
> 
> 
> --
> 
> Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#42400

Thanks for responding Terry.

I can assure you I did not initially realize both the `next` and the `__iter__` methods were implemented when I ran into my original problem. I saw a behavior and had to work backwards to realize why it was behaving the way it was (the comparison against Iterator). Once I realized this, the behavior made complete sense. It just dawned on me the fact that `next` was not being checked to be callable (I was surprised by this at the time) which is why I investigated the `Iterator.__subclasshook__` and assumed it was behaving incorrectly based on my assumptions.

On Sunday, March 31, 2013 3:47:07 PM UTC-4, Terry Jan Reedy wrote:
> On 3/31/2013 1:57 PM, Byron Ruth wrote:
> 
> > I submitted this as bug last night: http://bugs.python.org/issue17584 and was *honored* to be rejected by Raymond Hettinger. However, I would like feedback on whether my concern (this bug) is justified and clarity if not.
> 
> >
> 
> > Consider:
> 
> >
> 
> > ```python
> 
> > class A(object):
> 
> >      def __init__(self):
> 
> >          self.r = iter(range(5))
> 
> >      def __iter__(self):
> 
> >          return self
> 
> >      @property
> 
> >      def next(self):
> 
> >          return next(self.r)
> 
> > ```
> 
> >
> 
> > The `next` method is a property, however:
> 
> 
> 
> A competent Python programmer should not do that. In Py3, the method is 
> 
> properly renamed '__next__', which should make doing that accidentally 
> 
> even less likely.
> 
> 
> 
> >
> 
> > ```python
> 
> > from collections import Iterator
> 
> > a = A()
> 
> > isinstance(a, Iterator) # True
> 
> > next(a) # TypeError: 'int' object is not callable
> 
> > ```
> 
> >
> 
> > I am using `collections.Iterator` as the means to check if the object is an iterator,
> 
> 
> 
> Being an Iterator only means that it *might* be an iterator.
> 
> 
> 
>  > however I am not sure if that is _root_ problem here. My 
> 
> understanding of the iterator protocol is that is assumes the __iter__ 
> 
> and next *methods* are implemented. In the example, `A.next` is defined 
> 
> as a property, but is still identified as an iterator. To me, this is 
> 
> incorrect behavior since it's not conforming to the iterator protocol 
> 
> requirements (i.e. a `next` method, not a property).
> 
> 
> 
> There is more to any protocol than can be statically checked.
> 
> 
> 
> > Raymond stated: "The design of ABCs are to check for the existence to required named; none of them verify the signature."
> 
> 
> 
> Having the required attributes is currently the definition of being an 
> 
> instance of an ABC. Adding 'not a property' would be possible. but 
> 
> hardly worthwhile. Checking signatures would be worthwhile, but 
> 
> signatures are not yet available to Python for C-coded methods, let 
> 
> alone other implementations.
> 
> 
> 
>   I think I understand _why_ this is the case.. but I downstream 
> 
> libraries use `collections.Iterator` to determine if an object _is one_: 
> 
> see 
> 
> https://github.com/django/django/blob/master/django/utils/itercompat.py#L22-L31
> 
> >
> 
> > Who's job is it to check if `next` (and technically `__iter__`) are methods?
> 
> 
> 
> The programmer, and a user who does not trust the competence of the 
> 
> programmer. But this is the least of the possible errors.
> 
> 
> 
> --
> 
> Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#42435

On 03/31/2013 10:57 AM, Byron Ruth wrote:
> I submitted this as bug last night: http://bugs.python.org/issue17584 and was *honored* to be rejected by Raymond Hettinger. However, I would like feedback on whether my concern (this bug) is justified and clarity if not.
>
> Consider:
>
> ```python
> class A(object):
>      def __init__(self):
>          self.r = iter(range(5))
>      def __iter__(self):
>          return self
>      @property
>      def next(self):
>          return next(self.r)
> ```
>
> The `next` method is a property, however:
>
> ```python
> from collections import Iterator
> a = A()
> isinstance(a, Iterator) # True
> next(a) # TypeError: 'int' object is not callable
> ```
>
> I am using `collections.Iterator` as the means to check if the object is an iterator, however I am not sure if that is _root_ problem here. My understanding of the iterator protocol is that is assumes the __iter__ and next *methods* are implemented. In the example, `A.next` is defined as a property, but is still identified as an iterator. To me, this is incorrect behavior since it's not conforming to the iterator protocol requirements (i.e. a `next` method, not a property).

The root problem is that whoever implemented A did so incorrectly.


> Raymond stated: "The design of ABCs are to check for the existence to required named; none of them verify the signature." I think I understand _why_ this is the case.. but I downstream libraries use `collections.Iterator` to determine if an object _is one_: see https://github.com/django/django/blob/master/django/utils/itercompat.py#L22-L31
>
> Who's job is it to check if `next` (and technically `__iter__`) are methods?

The programmer who wrote it, with good tests.

Technically, the issue is not that 'next' is a property, but that it is not returning a callable.  If it did (and the 
callable was appropriate for the iterator), that would also work.

--
~Ethan~

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web