Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #87319 > unrolled thread

generator/coroutine terminology

Back to article view | Back to comp.lang.python

Contents

  generator/coroutine terminology Rustom Mody <rustompmody@gmail.com> - 2015-03-12 06:35 -0700
    Re: generator/coroutine terminology Chris Angelico <rosuav@gmail.com> - 2015-03-13 00:55 +1100
    Re: generator/coroutine terminology breamoreboy@gmail.com - 2015-03-12 06:57 -0700
    Re: generator/coroutine terminology Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-03-13 03:27 +1100
      Re: generator/coroutine terminology Rustom Mody <rustompmody@gmail.com> - 2015-03-12 09:52 -0700
        Re: generator/coroutine terminology Marko Rauhamaa <marko@pacujo.net> - 2015-03-12 19:55 +0200
          Re: generator/coroutine terminology Rustom Mody <rustompmody@gmail.com> - 2015-03-12 19:23 -0700
            Re: generator/coroutine terminology Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-03-13 14:30 +1100
              Re: generator/coroutine terminology Rustom Mody <rustompmody@gmail.com> - 2015-03-12 22:28 -0700
                Re: generator/coroutine terminology Chris Angelico <rosuav@gmail.com> - 2015-03-13 19:23 +1100
                  Re: generator/coroutine terminology Rustom Mody <rustompmody@gmail.com> - 2015-03-13 02:12 -0700
                    Re: generator/coroutine terminology Marko Rauhamaa <marko@pacujo.net> - 2015-03-13 11:36 +0200
                      Re: generator/coroutine terminology Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-03-14 17:04 +1100
                        Re: generator/coroutine terminology Marko Rauhamaa <marko@pacujo.net> - 2015-03-14 09:54 +0200
                          Re: generator/coroutine terminology Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-03-14 08:04 +0000
                            Re: generator/coroutine terminology Marko Rauhamaa <marko@pacujo.net> - 2015-03-14 10:30 +0200
                          Re: generator/coroutine terminology Ian Kelly <ian.g.kelly@gmail.com> - 2015-03-14 14:14 -0600
                            Re: generator/coroutine terminology Rustom Mody <rustompmody@gmail.com> - 2015-03-14 21:15 -0700
                          Re: generator/coroutine terminology Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-03-14 20:31 +0000
                        Re: generator/coroutine terminology Rustom Mody <rustompmody@gmail.com> - 2015-03-14 08:29 -0700
                          Re: generator/coroutine terminology Chris Angelico <rosuav@gmail.com> - 2015-03-15 02:56 +1100
                          Re: generator/coroutine terminology Rustom Mody <rustompmody@gmail.com> - 2015-03-14 08:59 -0700
                            Re: generator/coroutine terminology Chris Angelico <rosuav@gmail.com> - 2015-03-15 03:14 +1100
                              Re: generator/coroutine terminology Rustom Mody <rustompmody@gmail.com> - 2015-03-14 09:33 -0700
                                Re: generator/coroutine terminology Chris Angelico <rosuav@gmail.com> - 2015-03-15 03:51 +1100
                                  Re: generator/coroutine terminology Rustom Mody <rustompmody@gmail.com> - 2015-03-14 10:17 -0700
                                Re: generator/coroutine terminology Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-03-14 16:56 +0000
                                Re: generator/coroutine terminology Dave Angel <davea@davea.name> - 2015-03-14 13:07 -0400
                            Re: generator/coroutine terminology albert@spenarnc.xs4all.nl (Albert van der Horst) - 2015-03-31 12:57 +0000
                          Re: generator/coroutine terminology Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-03-15 19:37 +1100
                      Re: generator/coroutine terminology CHIN Dihedral <dihedral88888@gmail.com> - 2015-04-18 11:07 -0700
                    Re: generator/coroutine terminology Chris Angelico <rosuav@gmail.com> - 2015-03-13 22:32 +1100
        Re: generator/coroutine terminology Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2015-03-14 22:02 +0000
          Re: generator/coroutine terminology Marko Rauhamaa <marko@pacujo.net> - 2015-03-15 00:15 +0200
            Re: generator/coroutine terminology Chris Angelico <rosuav@gmail.com> - 2015-03-15 09:24 +1100
              Re: generator/coroutine terminology Marko Rauhamaa <marko@pacujo.net> - 2015-03-15 02:15 +0200
                Re: generator/coroutine terminology Chris Angelico <rosuav@gmail.com> - 2015-03-15 11:22 +1100
                  Re: generator/coroutine terminology Marko Rauhamaa <marko@pacujo.net> - 2015-03-15 02:48 +0200
                    Re: generator/coroutine terminology Chris Angelico <rosuav@gmail.com> - 2015-03-15 13:02 +1100
                Re: generator/coroutine terminology Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-03-16 12:03 +1100
                  Re: generator/coroutine terminology Marko Rauhamaa <marko@pacujo.net> - 2015-03-16 09:12 +0200
                    Re: generator/coroutine terminology Chris Angelico <rosuav@gmail.com> - 2015-03-16 18:21 +1100
                      Re: generator/coroutine terminology Marko Rauhamaa <marko@pacujo.net> - 2015-03-16 09:40 +0200
                        Re: generator/coroutine terminology Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-03-16 22:59 +1100
                    Re: generator/coroutine terminology Ian Kelly <ian.g.kelly@gmail.com> - 2015-03-16 01:37 -0600
                      Re: generator/coroutine terminology Marko Rauhamaa <marko@pacujo.net> - 2015-03-16 09:52 +0200
                        Re: generator/coroutine terminology Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-03-16 23:02 +1100
                          Re: generator/coroutine terminology Marko Rauhamaa <marko@pacujo.net> - 2015-03-16 14:42 +0200
                          Re: generator/coroutine terminology Jonas Wielicki <jonas@wielicki.name> - 2015-03-16 13:39 +0100
                    Re: generator/coroutine terminology Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-03-16 19:36 +1100
                      Re: generator/coroutine terminology Chris Angelico <rosuav@gmail.com> - 2015-03-16 19:58 +1100
                        Re: generator/coroutine terminology Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-03-16 22:51 +1100
                          Re: generator/coroutine terminology Chris Angelico <rosuav@gmail.com> - 2015-03-17 00:16 +1100
                      Re: generator/coroutine terminology Marko Rauhamaa <marko@pacujo.net> - 2015-03-16 14:32 +0200
                        Re: generator/coroutine terminology Rustom Mody <rustompmody@gmail.com> - 2015-03-16 05:51 -0700
                          Re: generator/coroutine terminology Marko Rauhamaa <marko@pacujo.net> - 2015-03-16 15:13 +0200
                            Re: generator/coroutine terminology Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-03-17 01:32 +1100
                              Re: generator/coroutine terminology Ian Kelly <ian.g.kelly@gmail.com> - 2015-03-16 08:45 -0600
                          Re: generator/coroutine terminology Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-03-17 00:39 +1100
                            Re: generator/coroutine terminology Rustom Mody <rustompmody@gmail.com> - 2015-03-16 07:19 -0700
                              Re: generator/coroutine terminology Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-03-16 14:26 +0000
                                Re: generator/coroutine terminology Rustom Mody <rustompmody@gmail.com> - 2015-03-16 07:37 -0700
                                  Re: generator/coroutine terminology Rustom Mody <rustompmody@gmail.com> - 2015-03-16 07:55 -0700
                                  Re: generator/coroutine terminology Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-03-16 18:19 +0000
                                    Re: generator/coroutine terminology Rustom Mody <rustompmody@gmail.com> - 2015-03-16 19:52 -0700
                                      Re: generator/coroutine terminology Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-03-17 03:07 +0000
                                        Re: generator/coroutine terminology Rustom Mody <rustompmody@gmail.com> - 2015-03-16 20:18 -0700
                                          Re: generator/coroutine terminology Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-03-17 03:25 +0000
                                            Re: generator/coroutine terminology Rustom Mody <rustompmody@gmail.com> - 2015-03-16 20:33 -0700
                                              Re: generator/coroutine terminology Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-03-17 03:55 +0000
                                      Re: generator/coroutine terminology Mario Figueiredo <marfig@gmail.com> - 2015-03-17 04:22 +0100
                              Re: generator/coroutine terminology Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-03-17 01:35 +1100
                                Re: generator/coroutine terminology Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-03-17 01:36 +1100
                            Re: generator/coroutine terminology Ian Kelly <ian.g.kelly@gmail.com> - 2015-03-16 08:52 -0600
                              Re: generator/coroutine terminology Marko Rauhamaa <marko@pacujo.net> - 2015-03-16 17:09 +0200
                                Re: generator/coroutine terminology Ian Kelly <ian.g.kelly@gmail.com> - 2015-03-16 09:26 -0600
                                  Re: generator/coroutine terminology Marko Rauhamaa <marko@pacujo.net> - 2015-03-16 18:05 +0200
                  Re: generator/coroutine terminology albert@spenarnc.xs4all.nl (Albert van der Horst) - 2015-03-31 13:18 +0000
                    Re: generator/coroutine terminology Dave Angel <davea@davea.name> - 2015-03-31 09:38 -0400
                      Re: generator/coroutine terminology albert@spenarnc.xs4all.nl (Albert van der Horst) - 2015-03-31 15:03 +0000
                        Re: generator/coroutine terminology Chris Angelico <rosuav@gmail.com> - 2015-04-01 02:36 +1100
                    Re: generator/coroutine terminology Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-04-03 17:02 +1100
                      Re: generator/coroutine terminology albert@spenarnc.xs4all.nl (Albert van der Horst) - 2015-04-18 17:52 +0000
                    Re: generator/coroutine terminology Paul Rubin <no.email@nospam.invalid> - 2015-04-02 23:46 -0700
    Re: generator/coroutine terminology Terry Reedy <tjreedy@udel.edu> - 2015-03-12 16:11 -0400
      Re: generator/coroutine terminology Marko Rauhamaa <marko@pacujo.net> - 2015-03-12 22:22 +0200

Page 3 of 5 — ← Prev page 1 2 [3] 4 5  Next page →

#87515

Steven D'Aprano <steve+comp.lang.python@pearwood.info>:

> Marko Rauhamaa wrote:
>> What features do generator iterators provide on top of generic
>> iterators?
>
> The biggest difference is syntactic. Here's an iterator which returns a
> never-ending sequence of squared numbers 1, 4, 9, 16, ...
>
> class Squares:
>     def __init__(self):
>         self.i = 0
>     def __next__(self):
>         self.i += 1
>         return self.i**2
>     def __iter__(self):
>         return self
>
>
> Here's the same thing written as a generator:
>
> def squares():
>     i = 1
>     while True:
>         yield i**2
>         i += 1

I was actually referring to the offered API. It still seems to me like
all iterators could offer close(), send() and throw(). Why? To make all
iterators drop-in replacements of each other.

Then, you could also stop wondering what to call the thingy returned by
a generator. Why, it would be an iterator. You wouldn't then have any
other use for a generator than the function that returns an iterator.

You could then decide if you want to reserve the name "generator" for
functions that contain a yield statement (syntactic notion) or call any
function returning an iterator a "generator" (semantic notion).


Marko

[toc] | [prev] | [next] | [standalone]

#87516

On Mon, Mar 16, 2015 at 6:12 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
>
> I was actually referring to the offered API. It still seems to me like
> all iterators could offer close(), send() and throw(). Why? To make all
> iterators drop-in replacements of each other.
>
> Then, you could also stop wondering what to call the thingy returned by
> a generator. Why, it would be an iterator. You wouldn't then have any
> other use for a generator than the function that returns an iterator.

That just adds unnecessary overhead to every iterator. Also, what
happens when you throw something into iter([1,2,3]) ? Or send it a
value? What happens?

ChrisA

[toc] | [prev] | [next] | [standalone]

#87521

Chris Angelico <rosuav@gmail.com>:

> On Mon, Mar 16, 2015 at 6:12 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
>>
>> I was actually referring to the offered API. It still seems to me like
>> all iterators could offer close(), send() and throw(). Why? To make all
>> iterators drop-in replacements of each other.
>
> [...]
>
> That just adds unnecessary overhead to every iterator.

Trivial implementations are a normal way to satisfy an API. That's why
/dev/null implements a close() method, for example.

> Also, what happens when you throw something into iter([1,2,3]) ? Or
> send it a value? What happens?

I would expect iter([1, 2, 3]) to behave highly analogously to
(x for x in [1, 2, 3]).

===begin /tmp/test.py===================================================
#!/usr/bin/env python3

def main():
    it = (x for x in [1, 2, 3])
    print(next(it))
    while True:
        try:
            print(it.send("hello"))
        except StopIteration:
            break

if __name__ == "__main__":
    main()
===end /tmp/test.py=====================================================

========================================================================
$ python3 /tmp/test.py
1
2
3
========================================================================

but:

===begin /tmp/test.py===================================================
#!/usr/bin/env python3

def main():
    it = iter([1, 2, 3])
    print(next(it))
    while True:
        try:
            print(it.send("hello"))
        except StopIteration:
            break

if __name__ == "__main__":
    main()
===end /tmp/test.py=====================================================

========================================================================
$ python3 /tmp/test.py
1
Traceback (most recent call last):
  File "/tmp/test.py", line 13, in <module>
    main()
  File "/tmp/test.py", line 8, in main
    print(it.send("hello"))
AttributeError: 'list_iterator' object has no attribute 'send'
========================================================================


Marko

[toc] | [prev] | [next] | [standalone]

#87545

Marko Rauhamaa wrote:

> Chris Angelico <rosuav@gmail.com>:
> 
>> On Mon, Mar 16, 2015 at 6:12 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
>>>
>>> I was actually referring to the offered API. It still seems to me like
>>> all iterators could offer close(), send() and throw(). Why? To make all
>>> iterators drop-in replacements of each other.
>>
>> [...]
>>
>> That just adds unnecessary overhead to every iterator.
> 
> Trivial implementations are a normal way to satisfy an API. That's why
> /dev/null implements a close() method, for example.

Absolutely. But you''ll notice it doesn't implement upper() and lower()
methods, because it isn't a string, it's a file.

Iterators shouldn't implement send() etc. methods because they aren't
coroutines.


>> Also, what happens when you throw something into iter([1,2,3]) ? Or
>> send it a value? What happens?
> 
> I would expect iter([1, 2, 3]) to behave highly analogously to
> (x for x in [1, 2, 3]).

I believe that it is an unfortunate mistake that generators which aren't
coroutines support send. At the very least it is an ugly design wart, at
worst it is an outright bug.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#87520

On Mon, Mar 16, 2015 at 1:12 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> I was actually referring to the offered API. It still seems to me like
> all iterators could offer close(), send() and throw(). Why? To make all
> iterators drop-in replacements of each other.

The purpose of close, send and throw is to implement *coroutines*, not
iterators. The fact that coroutines in Python happen to be implemented
as a type of iterator doesn't mean that all iterators need to have
them. You're free to add these methods to non-generator iterators that
you write in order to create iterators that pretend to be coroutines,
but I'm having a hard time seeing what purpose would be served by
this.

If on the other hand these methods were to be added to the iterator
protocol, it would just create unnecessary implementation overhead for
the 99.99% of non-generator iterators that are under no illusions
about the fact that they aren't coroutines.

[toc] | [prev] | [next] | [standalone]

#87523

Ian Kelly <ian.g.kelly@gmail.com>:

> If on the other hand these methods were to be added to the iterator
> protocol, it would just create unnecessary implementation overhead for
> the 99.99% of non-generator iterators that are under no illusions
> about the fact that they aren't coroutines.

Why should

   (x for x in [1, 2, 3])

be a valid coroutine but

   iter([1, 2, 3])

not?

Anyway, calling close() on an iterator can be a necessary requirement
even in ordinary generator usage. Only now they have to know if the
underlying iterator was sired by a generator or some other constructor.


Marko

[toc] | [prev] | [next] | [standalone]

#87546

Marko Rauhamaa wrote:

> Anyway, calling close() on an iterator can be a necessary requirement
> even in ordinary generator usage. Only now they have to know if the
> underlying iterator was sired by a generator or some other constructor.

Can you give an actual example of that?

(To be honest, I'm not even sure what the use-case for close() on coroutines
is in the first place. If you don't want to send any more items into it,
just don't send any more items into it.)


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#87548

Steven D'Aprano <steve+comp.lang.python@pearwood.info>:

> (To be honest, I'm not even sure what the use-case for close() on
> coroutines is in the first place. If you don't want to send any more
> items into it, just don't send any more items into it.)

Without close(), you might leave resources hanging. See PEP 325:

    Rejected in favor of PEP 342 which includes substantially all of the
    requested behavior in a more refined form.

And PEP 342:

    Raymond Hettinger (PEP 288) and Samuele Pedroni (PEP 325) first
    formally proposed the ideas of communicating values or exceptions
    into generators, and the ability to "close" generators.

One *might* take the stance that resource allocation is outside the
scope of iterators, generator iterators and coroutine generator
iterators. As it stands, it is unclear if the application *must* call
close() on generator iterators that are left hanging.


Marko

[toc] | [prev] | [next] | [standalone]

#87549

[Multipart message — attachments visible in raw view] — view raw

On 16.03.2015 13:02, Steven D'Aprano wrote:
> (To be honest, I'm not even sure what the use-case for close() on coroutines
> is in the first place. If you don't want to send any more items into it,
> just don't send any more items into it.)

Just like with file-likes, it is useful to clean up resources. If you
use a coroutine with the lxml.etree.xmlfile interface [1] for instance
(code quoted from the reference):

    def writer(out_stream):
        with xmlfile(out_stream) as xf:
             with xf.element('{http://etherx.jabber.org/streams}stream'):
                  try:
                      while True:
                          el = (yield)
                          xf.write(el)
                          xf.flush()
                  except GeneratorExit:
                      pass

This allows controlled (i.e. not governed by garbage collection) and
clean shutdown of a coroutine (albeit I’m not sure why they catch the
GeneratorExit here), which will close the opened <stream> tag.

regards,
jwi

   [1]: http://lxml.de/api.html#incremental-xml-generation

[toc] | [prev] | [next] | [standalone]

#87533

On Monday 16 March 2015 18:12, Marko Rauhamaa wrote:

> Steven D'Aprano <steve+comp.lang.python@pearwood.info>:
> 
>> Marko Rauhamaa wrote:
>>> What features do generator iterators provide on top of generic
>>> iterators?
>>
>> The biggest difference is syntactic. Here's an iterator which returns a
>> never-ending sequence of squared numbers 1, 4, 9, 16, ...
>>
>> class Squares:
>>     def __init__(self):
>>         self.i = 0
>>     def __next__(self):
>>         self.i += 1
>>         return self.i**2
>>     def __iter__(self):
>>         return self
>>
>>
>> Here's the same thing written as a generator:
>>
>> def squares():
>>     i = 1
>>     while True:
>>         yield i**2
>>         i += 1
> 
> I was actually referring to the offered API. It still seems to me like
> all iterators could offer close(), send() and throw(). Why? To make all
> iterators drop-in replacements of each other.

How could these two iterators *possibly* be drop in replacements for each 
other in any real body of code?


def inorder(tree):
    for node in inorder(tree.left):
        yield node.payload
    yield tree.payload
    for node in inorder(tree.right):
        yield node.payload


def clean_up(collection, condition, cleaner=None):
    if cleaner is None:
        def cleaner(obj):
            try:
                obj.cache = {}
                obj.ref = None
            except Exception:
                return False
            return True
    for i, item in enumerate(collection):
        if condition(item):
            flag = cleaner(item)
            if flag:
                yield Warn(MSG % (i, item))


You can't just substitute any random iterator for another, any more than you 
can substitute any random two functions for each other.

Overly pure (in the comp sci theory sense) interfaces can be silly. Just 
because the Soyez rocket and my nephew's tricycle are both instances of 
Vehicle doesn't mean that I should be able to substitute one for the other. 
OOP theory says they should. Reality says No Way.

Why should I, the class designer, be forced to offer send(), throw() and 
close() methods on my iterators if I'm never going to use them as 
coroutines? If I .send() into either of the generators above, its a 
conceptual mistake and I should get an error. The fact that I don't is an 
implementation detail, and one which hopefully will be fixed. I expect that 
the interpreter can tell the difference between 

    yield spam

and 

    x = yield spam

and only allow send(), close() and throw() on generators which include the 
second form. If it can't, then that's a limitation, not a feature to be 
emulated.


> Then, you could also stop wondering what to call the thingy returned by
> a generator. 

Apart from a few die-hards who are hoping for some sort of Académie 
Française to define the One True meaning of words, most of us stopped 
wondering what to call the thingy returned by a generator function years 
ago. It's a generator, just like introspection of the object tells you.


> Why, it would be an iterator. You wouldn't then have any
> other use for a generator than the function that returns an iterator.

Lots of things are iterators. Doesn't make them generators.


> You could then decide if you want to reserve the name "generator" for
> functions that contain a yield statement (syntactic notion) or call any
> function returning an iterator a "generator" (semantic notion).

Why would you want the second? Do you have special names for {all functions 
which return strings} and {all functions which return bools}?

Functions are normally described by their *form* or *purpose*, not their 
return result, e.g. inner function, public function, callback function, 
reentrant function, filter function.


-- 
Steve

[toc] | [prev] | [next] | [standalone]

#87538

On Mon, Mar 16, 2015 at 7:36 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> I expect that the interpreter can tell the difference between
>
>     yield spam
>
> and
>
>     x = yield spam
>
> and only allow send(), close() and throw() on generators which include the
> second form. If it can't, then that's a limitation, not a feature to be
> emulated.
>

Hmm. That would imply that an expression is different if it's used
somewhere. In Python, there's no difference between these two function
calls:

x = func(1,2,3)
func(1,2,3)

except for the actual name binding. If a yield expression enabled
send() if and only if its return value were used, then it would be
extremely surprising:

def use_and_discard():
    while True:
        _ = (yield 1)

def just_yield():
    while True:
        yield 1

So... I can send into the first but not into the second?

That said, though, it would be quite reasonable for a *linter* to warn
you about sending into a generator that never uses sent values. With a
good type inference system (not sure if MyPy is sufficient here), it
would be possible to distinguish between those two functions and
declare that the first one has the return type "sendable_generator"
and the second one "nonsendable_generator", and give a warning if you
send into the latter. But I don't think that's the language's job.

ChrisA

[toc] | [prev] | [next] | [standalone]

#87543

Chris Angelico wrote:

> On Mon, Mar 16, 2015 at 7:36 PM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> I expect that the interpreter can tell the difference between
>>
>>     yield spam
>>
>> and
>>
>>     x = yield spam
>>
>> and only allow send(), close() and throw() on generators which include
>> the second form. If it can't, then that's a limitation, not a feature to
>> be emulated.
>>
> 
> Hmm. That would imply that an expression is different if it's used
> somewhere. In Python, there's no difference between these two function
> calls:
> 
> x = func(1,2,3)
> func(1,2,3)
> 
> except for the actual name binding.

That's a pretty big difference though :-)

Just for the record, it's not just name bindings that make a generator
behave as a coroutine. E.g.:

py> def co():
...     print( (yield 1) + 2 )
...
py> a = co()
py> next(a)
1
py> a.send(98)
100
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration


Not the most *useful* example, I grant, but it demonstrates the idea.
However, for simplicity let's ignore that and pretend it's only name
binding we care about.


> If a yield expression enabled 
> send() if and only if its return value were used, then it would be
> extremely surprising:

It's surprising to me that you can send() into a non-coroutine and have it
ignored. I'm not even sure that is documented behaviour or just an accident
of implementation.

 
> def use_and_discard():
>     while True:
>         _ = (yield 1)
> 
> def just_yield():
>     while True:
>         yield 1
> 
> So... I can send into the first but not into the second?

That's the idea.

The second one is not a coroutine, and cannot be used as a coroutine in any
useful way. I maintain that it is a mistake that you can send() into it at
all, and if somebody does, it is invariably an error. That would be like
send()ing into None, or a str. We know what the Zen says about errors...


> That said, though, it would be quite reasonable for a *linter* to warn
> you about sending into a generator that never uses sent values. With a
> good type inference system (not sure if MyPy is sufficient here), it
> would be possible to distinguish between those two functions and
> declare that the first one has the return type "sendable_generator"
> and the second one "nonsendable_generator", and give a warning if you
> send into the latter. But I don't think that's the language's job.

I do :-)

I think it would be hard for a linter to do this. It can't just do a
type-check, because the types of generator-iterators and
generator-coroutines are the same. It would need to actually analyse the
source code, AST, or byte-code. That's doable, but the compiler already
does that, and compiles different code for the two cases:

py> from dis import dis
py> def g1():
...     yield 1
...
py> def g2():
...     x = yield 1
...
py> dis(g1)
  2           0 LOAD_CONST               1 (1)
              3 YIELD_VALUE
              4 POP_TOP
              5 LOAD_CONST               0 (None)
              8 RETURN_VALUE
py> dis(g2)
  2           0 LOAD_CONST               1 (1)
              3 YIELD_VALUE
              4 STORE_FAST               0 (x)
              7 LOAD_CONST               0 (None)
             10 RETURN_VALUE


so it should be easy for the compiler to recognise a yield used as if it
were a statement versus one used as an expression, and take appropriate
steps. But maybe there are subtle corner cases I haven't thought of.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#87554

On Mon, Mar 16, 2015 at 10:51 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Chris Angelico wrote:
> Just for the record, it's not just name bindings that make a generator
> behave as a coroutine. E.g.:
>
> py> def co():
> ...     print( (yield 1) + 2 )
> ...
> py> a = co()
> py> next(a)
> 1
> py> a.send(98)
> 100
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> StopIteration

Yep, I agree. It's not the name binding, it's whether the yield
expression's result is instantly discarded vs used in some way.

> It's surprising to me that you can send() into a non-coroutine and have it
> ignored. I'm not even sure that is documented behaviour or just an accident
> of implementation.

I agree conceptually; but there are two different consistencies here,
and they clash, my lords, they clash! One is that the API for a
coroutine includes send() but the API for a non-coroutine generator
doesn't, and that trying to send() into the latter should raise an
error. (Side point: What error? I'd accept AttributeError, but since
it's possible to mix sendable and nonsendable yields, it might be
better to have it raise at the time of the call - that is, when you
send a non-None value into a generator that's just going to discard
the value, it raises on arrival. In that case, possibly ValueError?
GeneratorStyleError?) And on the other hand, we have a very strong
Python convention that any expression can be used as a statement
without altering the effect of that expression in any way. Assigning
something to a name that you never use, or passing it as an argument
to a no-op function, should not suddenly change the behaviour of
anything.

>> That said, though, it would be quite reasonable for a *linter* to warn
>> you about sending into a generator that never uses sent values. With a
>> good type inference system (not sure if MyPy is sufficient here), it
>> would be possible to distinguish between those two functions and
>> declare that the first one has the return type "sendable_generator"
>> and the second one "nonsendable_generator", and give a warning if you
>> send into the latter. But I don't think that's the language's job.
>
> I do :-)
>
> I think it would be hard for a linter to do this. It can't just do a
> type-check, because the types of generator-iterators and
> generator-coroutines are the same. It would need to actually analyse the
> source code, AST, or byte-code. That's doable, but the compiler already
> does that, and compiles different code for the two cases:
>
> py> from dis import dis
> py> def g1():
> ...     yield 1
> ...
> py> def g2():
> ...     x = yield 1
> ...
> py> dis(g1)
>   2           0 LOAD_CONST               1 (1)
>               3 YIELD_VALUE
>               4 POP_TOP
>               5 LOAD_CONST               0 (None)
>               8 RETURN_VALUE
> py> dis(g2)
>   2           0 LOAD_CONST               1 (1)
>               3 YIELD_VALUE
>               4 STORE_FAST               0 (x)
>               7 LOAD_CONST               0 (None)
>              10 RETURN_VALUE
>
>
> so it should be easy for the compiler to recognise a yield used as if it
> were a statement versus one used as an expression, and take appropriate
> steps. But maybe there are subtle corner cases I haven't thought of.

That code isn't actually all that different, though. What you have is this:

1) Load up a value
2) Yield, which pops the value off, yields it, and pushes the sent value or None
3a) in g1: Discard the value of the expression that we've just finished
3b) in g2: Store the value of that expression into x
4) Load None, and return it.

Steps 1, 3, and 4 would all be identical if you replace the "yield 1"
with, say, "print(1)". It'll pop the 1 off, do something, and leave a
result on the stack. The difference between assignment and
non-assignment comes later.

The risk I see here is that a simple optimization might suddenly make
a semantic change to something. Imagine this function:

def gen():
    while True:
        x = (yield 1)
        if x == 5: break

Okay, so it cares about sent values. Well and good. Then we change it
so it doesn't always care:

def gen(state):
    while True:
        x = (yield 1)
        if state and x == 5: break

Ahh but look! That's a mode-switch parameter. We should break that out
into two functions.

def gen_stoppable():
    while True:
        x = (yield 1)
        if x == 5: break

def gen_unstoppable():
    while True:
        x = (yield 1)

Now, according to the current rules, you can change that last line to
just "yield 1" without the two functions differing in API. There's no
sent value that will stop the second one, but it will accept and
ignore sent values, just as the first one does. (Imagine that's some
sort of password, and the second generator is for the case where the
account is locked and there IS no password.) But according to your
rules, changing it to no longer assign the yielded value somewhere
would make a semantic and API change to the function. That seems, to
me, at least as surprising as the case of send() working when the
value's going to be ignored.

Either way, it's not going to be ideal. I do hear you about the
problem of sending when you shouldn't be able to... but I'm not
convinced the cure isn't worse than the disease.

ChrisA

[toc] | [prev] | [next] | [standalone]

#87547

Steven D'Aprano <steve+comp.lang.python@pearwood.info>:

> If I .send() into either of the generators above, its a conceptual
> mistake and I should get an error. The fact that I don't is an
> implementation detail, and one which hopefully will be fixed.

So what we have now is:

  (1) plain iterators

  (2) generator iterators

  (3) coroutine generator iterators

At the moment (2) and (3) are unified, which you don't like. (1) and (2)
are not unified, which I don't like.

You and I can pretend (2) didn't have send(), throw() and close() and be
satisfied. Then, generator iterators are just plain iterators from the
point of view of the application programmer.

However, it seems to me close() is intended to be used for (2) as well.
The way I'm reading PEP 342, it seems to encompass the objectives of the
rejected PEP 325  ("Resource-Release Support for Generators"):


Marko

[toc] | [prev] | [next] | [standalone]

#87550

On Monday, March 16, 2015 at 6:02:48 PM UTC+5:30, Marko Rauhamaa wrote:
> So what we have now is:
> 
>   (1) plain iterators
> 
>   (2) generator iterators
> 
>   (3) coroutine generator iterators
> 

> (1) and (2) are not unified, which I don't like.

Can you expand on that a bit?

It may help to read this 15 year old thread starting
https://mail.python.org/pipermail/python-dev/2001-June/015478.html
to see how many python stalwarts dont like the current state of affairs

[toc] | [prev] | [next] | [standalone]

#87553

Rustom Mody <rustompmody@gmail.com>:

> On Monday, March 16, 2015 at 6:02:48 PM UTC+5:30, Marko Rauhamaa wrote:
>> So what we have now is:
>>   (1) plain iterators
>>   (2) generator iterators
>>   (3) coroutine generator iterators
>
>> (1) and (2) are not unified, which I don't like.
>
> Can you expand on that a bit?

(2) contains methods, most notably close(), that (1) does not provide.
If I get an iterator from a black box source, I don't know if I'm
allowed/supposed to call close() on it.


Marko

[toc] | [prev] | [next] | [standalone]

#87561

On Tue, 17 Mar 2015 12:13 am, Marko Rauhamaa wrote:

> If I get an iterator from a black box source, I don't know if I'm
> allowed/supposed to call close() on it.

In no particular order:

#1
if hasattr(some_iterator, 'close'):
    some_iterator.close()


#2
close = getattr(some_iterator, 'close', None)
if close is not None:
    close()


#3
try:
    close = some_iterator.close
except AttributeError:
    pass
else:
    close()


#4 (The most pythonic solution of all.) 
If you want to call close(), for some reason, *make it part of your API* and
insist that whatever object is passed to you supports close(), or else it
is an error.




-- 
Steven

[toc] | [prev] | [next] | [standalone]

#87565

On Mon, Mar 16, 2015 at 8:32 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Tue, 17 Mar 2015 12:13 am, Marko Rauhamaa wrote:
>
>> If I get an iterator from a black box source, I don't know if I'm
>> allowed/supposed to call close() on it.
>
> In no particular order:
>
> #1
> if hasattr(some_iterator, 'close'):
>     some_iterator.close()
>
>
> #2
> close = getattr(some_iterator, 'close', None)
> if close is not None:
>     close()
>
>
> #3
> try:
>     close = some_iterator.close
> except AttributeError:
>     pass
> else:
>     close()

Note that these fail if some_iterator is a file-like object. Those
have close methods, but as part of the file api, not the iterator api,
and so the owner of the object is probably not expecting you to close
it for them.

[toc] | [prev] | [next] | [standalone]

#87557

On Mon, 16 Mar 2015 11:51 pm, Rustom Mody wrote:

> It may help to read this 15 year old thread starting
> https://mail.python.org/pipermail/python-dev/2001-June/015478.html
> to see how many python stalwarts dont like the current state of affairs

/s/dont/didn't/

That's a fifteen year old thread, written before generators were introduced.
Guido's intuition has been vindicated -- generators have proven to be a
great and powerful feature, and the reuse of "def" for both generator
functions and regular functions has turned out to be no more confusing in
practice than the use of "def" for both functions and methods[1].


The argument is that generator-producers (def-with-yield) are not like
regular functions because calling the def-with-yield doesn't execute the
code inside them. Given three objects:

def f(): return 1

def g(): yield 1

class C: ...

the argument goes that:

* Calling f() executes the code inside f.

* Calling g() *doesn't* execute the code inside f, but runs 
  some initialisation code and returns an instance; it's the
  instance that executes the code inside f, using a special
  method.

* Calling C() also doesn't execute the code inside C, but 
  runs some initialisation code and returns an instance.


Therefore def-with-yield is closer to a class than a function and should not
reuse the same keyword. Or so the argument goes.

But this argument doesn't take into account the way generators are used.
They are used like functions, not like instances[2] like this:

# No
instance = gen()
instance.fe()
instance.fi()
instance.fo()
instance.fum()


but like functions, like this:

# Yes
for x in gen(): ...
sum(gen())
result = map(func, gen())


Here's a pure-Python version of map from Python 2:

def map(func, iterable):
     accum = []
     for item in iterable:
         accum.append(func(item))
     return accum

Here's the same thing for the Python 3 iterator version:

def map(func, iterable):
     for item in iterable:
         yield func(item)


The similarities when writing these are undeniable. Drop the accumulator,
replace any lines that store values into the accumulator with a yield, and
drop the final return. The way we write and use generators is closer to the
way we write and use functions than classes. If we rule out introducing a
new keyword, and insist on picking either "def" or "class", I think it is
obvious that "def" is the better choice and "class" would be completely
inappropriate. And of course, from a comp science theoretic perspective,
generators are a kind of subroutine, not a kind of type.




[1] Methods are, of course, actually just functions under the hood. The
conversion to MethodType is done at method-lookup time, by the descriptor
protocol.

[2] I'm aware that they are instances. You know what I mean.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#87558

On Monday, March 16, 2015 at 7:10:03 PM UTC+5:30, Steven D'Aprano wrote:
> And of course, from a comp science theoretic perspective,
> generators are a kind of subroutine, not a kind of type.

You just showed Marko a few posts back, that
A generator is a special case of an iterator.
And you wrote the iterator with 'class'.
And from python 2.2(?) classes are types.

So I dont know what "comp science theoretic perspective" you are describing...

From mine:

The generator
def gen():
  yield 1
  yield 2

is much closer to the list (ie data) [1,2]
than to say

def foo():
  print 1
  print 2

The only difference is that the list memoizes the data whereas the generator doesn't.

CLU perspective:
The iterator for a collection of complex_numbers can be used interchangeably with that for an array of integers
from https://en.wikipedia.org/wiki/CLU_%28programming_language%29
[Note: CLU-iterator ≡ python-generator]

Haskell perspective: 
Lists are by default lazy and memoized.
IOW in Haskell: List ≡ Lazy list ≡ python generator + memoization

Scheme perspective:
Programmer can choose between normal and lazy lists.
Lazy lists are just normal lists + delay/force where
delay x ≡ lambda: x
force x ≡ x()

======================
Anyways...

Yes 15 years are past.
I dont expect the def can be revoked now.
[As far as I am concerned its a minor wart]
But the mess around the docs can certainly be cleaned up.

[toc] | [prev] | [next] | [standalone]

Page 3 of 5 — ← Prev page 1 2 [3] 4 5  Next page →

Back to top | Article view | comp.lang.python

csiph-web