Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #102092 > unrolled thread

Re: Question about asyncio and blocking operations

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Question about asyncio and blocking operations Ian Kelly <ian.g.kelly@gmail.com> - 2016-01-25 08:45 -0700
    Re: Question about asyncio and blocking operations Rustom Mody <rustompmody@gmail.com> - 2016-01-25 22:01 -0800
      Re: Question about asyncio and blocking operations Marko Rauhamaa <marko@pacujo.net> - 2016-01-26 09:23 +0200
        Re: Question about asyncio and blocking operations Paul Rubin <no.email@nospam.invalid> - 2016-01-25 23:53 -0800

#102092 — Re: Question about asyncio and blocking operations

On Mon, Jan 25, 2016 at 8:32 AM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
>
> On Jan 25, 2016 2:04 AM, "Frank Millman" <frank@chagford.com> wrote:
>>
>> "Ian Kelly"  wrote in message
>> news:CALwzidnGOgpx+CpMVBA8vpEFuq4-BwMVS0gZ3ShB0oWZi0Bw+Q@mail.gmail.com...
>>>
>>> This seems to be a common misapprehension about asyncio programming.
>>> While coroutines are the focus of the library, they're based on
>>> futures, and so by working at a slightly lower level you can also
>>> handle them as such. So  while this would be the typical way to use
>>> run_in_executor:
>>>
>>> async def my_coroutine(stuff):
>>>     value = await get_event_loop().run_in_executor(None,
>>> blocking_function, stuff)
>>>     result = await do_something_else_with(value)
>>>     return result
>>>
>>> This is also a perfectly valid way to use it:
>>>
>>> def normal_function(stuff):
>>>     loop = get_event_loop()
>>>     coro = loop.run_in_executor(None, blocking_function, stuff)
>>>     task = loop.create_task(coro)
>>>     task.add_done_callback(do_something_else)
>>>     return task
>>
>>
>> I am struggling to get my head around this.
>>
>> 1. In the second function, AFAICT coro is already a future. Why is it
>> necessary to turn it into a task? In fact when I tried that in my testing, I
>> got an assertion error -
>>
>> File: "C:\Python35\lib\asyncio\base_events.py", line 211, in create_task
>>    task = tasks.Task(coro, loop=self)
>> File: "C:\Python35\lib\asyncio\tasks.py", line 70, in __init__
>>    assert coroutines.iscoroutine(coro), repr(coro)
>> AssertionError: <Future pending ... >
>
> I didn't test this; it was based on the documentation, which says that
> run_in_executor is a coroutine. Looking at the source, it's actually a
> function that returns a future, so this may be a documentation bug.

And now I'm reminded of this note in the asyncio docs:

"""
Note: In this documentation, some methods are documented as
coroutines, even if they are plain Python functions returning a
Future. This is intentional to have a freedom of tweaking the
implementation of these functions in the future. If such a function is
needed to be used in a callback-style code, wrap its result with
ensure_future().
"""

IMO such methods should simply be documented as awaitables, not
coroutines. I wonder if that's already settled, or if it's worth
starting a discussion around.

[toc] | [next] | [standalone]

#102105

On Monday, January 25, 2016 at 9:16:13 PM UTC+5:30, Ian wrote:
> On Mon, Jan 25, 2016 at 8:32 AM, Ian Kelly wrote:
> >
> > On Jan 25, 2016 2:04 AM, "Frank Millman"  wrote:
> >>
> >> "Ian Kelly"  wrote in message
> >>>
> >>> This seems to be a common misapprehension about asyncio programming.
> >>> While coroutines are the focus of the library, they're based on
> >>> futures, and so by working at a slightly lower level you can also
> >>> handle them as such. So  while this would be the typical way to use
> >>> run_in_executor:
> >>>
> >>> async def my_coroutine(stuff):
> >>>     value = await get_event_loop().run_in_executor(None,
> >>> blocking_function, stuff)
> >>>     result = await do_something_else_with(value)
> >>>     return result
> >>>
> >>> This is also a perfectly valid way to use it:
> >>>
> >>> def normal_function(stuff):
> >>>     loop = get_event_loop()
> >>>     coro = loop.run_in_executor(None, blocking_function, stuff)
> >>>     task = loop.create_task(coro)
> >>>     task.add_done_callback(do_something_else)
> >>>     return task
> >>
> >>
> >> I am struggling to get my head around this.
> >>
> >> 1. In the second function, AFAICT coro is already a future. Why is it
> >> necessary to turn it into a task? In fact when I tried that in my testing, I
> >> got an assertion error -
> >>
> >> File: "C:\Python35\lib\asyncio\base_events.py", line 211, in create_task
> >>    task = tasks.Task(coro, loop=self)
> >> File: "C:\Python35\lib\asyncio\tasks.py", line 70, in __init__
> >>    assert coroutines.iscoroutine(coro), repr(coro)
> >> AssertionError: <Future pending ... >
> >
> > I didn't test this; it was based on the documentation, which says that
> > run_in_executor is a coroutine. Looking at the source, it's actually a
> > function that returns a future, so this may be a documentation bug.
> 
> And now I'm reminded of this note in the asyncio docs:
> 
> """
> Note: In this documentation, some methods are documented as
> coroutines, even if they are plain Python functions returning a
> Future. This is intentional to have a freedom of tweaking the
> implementation of these functions in the future. If such a function is
> needed to be used in a callback-style code, wrap its result with
> ensure_future().
> """
> 
> IMO such methods should simply be documented as awaitables, not
> coroutines. I wonder if that's already settled, or if it's worth
> starting a discussion around.

Bah -- What a bloody mess!
And thanks for pointing this out, Ian.
Keep wondering whether my brain is atrophying, or its rocket science or...

[toc] | [prev] | [next] | [standalone]

#102106

Rustom Mody <rustompmody@gmail.com>:

> Bah -- What a bloody mess!
> And thanks for pointing this out, Ian.
> Keep wondering whether my brain is atrophying, or its rocket science or...

I'm afraid the asyncio idea will not fly.

Adding the keywords "async" and "await" did make things much better, but
the programming model seems very cumbersome.

Say you have an async that calls a nonblocking function as follows:


     async def t():
         ...
         f()
         ...

     def f():
         ...
         g()
         ...

     def g():
         ...
         h()
         ...

     def h():
         ...

Then, you need to add a blocking call to h(). You then have a cascading
effect of having to sprinkle asyncs and awaits everywhere:

     async def t():
         ...
         await f()
         ...

     async def f():
         ...
         await g()
         ...

     async def g():
         ...
         await h()
         ...

     async def h():
         ...
         await ...
         ...

A nasty case of nonlocality. Makes you wonder if you ought to declare
*all* functions *always* as asyncs just in case they turn out that way.

Note that neither the multithreading model (which I dislike) nor the
callback hell (which I like) suffer from this problem.


Marko

[toc] | [prev] | [next] | [standalone]

#102107

Marko Rauhamaa <marko@pacujo.net> writes:
> Note that neither the multithreading model (which I dislike) nor the
> callback hell (which I like) suffer from this problem.

There are some runtimes (GHC and Erlang) where everything is nonblocking
under the covers, which lets even the asyncs be swept under the rug.
Similarly with some low-tech cooperative multitaskers, say in Forth.
When you've got a mixture of blocking and nonblocking, it becomes a
mess.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web