Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: "Frank Millman" <frank@chagford.com>
Newsgroups: comp.lang.python
Subject: Re: Question about asyncio and blocking operations
Date: Mon, 25 Jan 2016 11:02:40 +0200
Lines: 62
Message-ID: <mailman.216.1453712587.15297.python-list@python.org>
References: <n8038j$575$1@ger.gmane.org> <CALwzidnGOgpx+CpMVBA8vpEFuq4-BwMVS0gZ3ShB0oWZi0Bw+Q@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original
Content-Transfer-Encoding: 7bit
In-Reply-To: <CALwzidnGOgpx+CpMVBA8vpEFuq4-BwMVS0gZ3ShB0oWZi0Bw+Q@mail.gmail.com>
Importance: Normal
Precedence: list
Xref: csiph.com comp.lang.python:102088

"Ian Kelly"  wrote in message 
news:CALwzidnGOgpx+CpMVBA8vpEFuq4-BwMVS0gZ3ShB0oWZi0Bw+Q@mail.gmail.com...
>
> On Sat, Jan 23, 2016 at 7:38 AM, Frank Millman <frank@chagford.com> wrote:
> > Here is the difficulty. The recommended way to handle a blocking 
> > operation
> > is to run it as task in a different thread, using run_in_executor(). 
> > This
> > method is a coroutine. An implication of this is that any method that 
> > calls
> > it must also be a coroutine, so I end up with a chain of coroutines
> > stretching all the way back to the initial event that triggered it.
>
> This seems to be a common misapprehension about asyncio programming.
> While coroutines are the focus of the library, they're based on
> futures, and so by working at a slightly lower level you can also
> handle them as such. So  while this would be the typical way to use
> run_in_executor:
>
> async def my_coroutine(stuff):
>     value = await get_event_loop().run_in_executor(None,
> blocking_function, stuff)
>     result = await do_something_else_with(value)
>     return result
>
> This is also a perfectly valid way to use it:
>
> def normal_function(stuff):
>     loop = get_event_loop()
>     coro = loop.run_in_executor(None, blocking_function, stuff)
>     task = loop.create_task(coro)
>     task.add_done_callback(do_something_else)
>     return task

I am struggling to get my head around this.

1. In the second function, AFAICT coro is already a future. Why is it 
necessary to turn it into a task? In fact when I tried that in my testing, I 
got an assertion error -

File: "C:\Python35\lib\asyncio\base_events.py", line 211, in create_task
    task = tasks.Task(coro, loop=self)
File: "C:\Python35\lib\asyncio\tasks.py", line 70, in __init__
    assert coroutines.iscoroutine(coro), repr(coro)
AssertionError: <Future pending ... >

2. In the first function, calling 'run_in_executor' unblocks the main loop 
so that it can continue with other tasks, but the function itself is 
suspended until the blocking function returns. In the second function, I 
cannot see how the function gets suspended. It looks as if the blocking 
function will run in the background, and the main function will continue.

I would like to experiment with this further, but I would need to see the 
broader context - IOW see the 'caller' of normal_function(), and see what it 
does with the return value.

I feel I am getting closer to an 'aha' moment, but I am not there yet, so 
all info is appreciated.

Frank