Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #103074 > unrolled thread

How the heck does async/await work in Python 3.5

Back to article view | Back to comp.lang.python

Contents

  How the heck does async/await work in Python 3.5 Mark Lawrence <breamoreboy@yahoo.co.uk> - 2016-02-17 22:08 +0000
    Re: How the heck does async/await work in Python 3.5 Steven D'Aprano <steve@pearwood.info> - 2016-02-20 13:36 +1100
      Re: How the heck does async/await work in Python 3.5 Rustom Mody <rustompmody@gmail.com> - 2016-02-19 21:24 -0800
        Re: How the heck does async/await work in Python 3.5 Rustom Mody <rustompmody@gmail.com> - 2016-02-19 21:34 -0800
          Re: How the heck does async/await work in Python 3.5 Paul Rubin <no.email@nospam.invalid> - 2016-02-19 22:44 -0800
            Re: How the heck does async/await work in Python 3.5 Steven D'Aprano <steve@pearwood.info> - 2016-02-21 18:17 +1100
              Re: How the heck does async/await work in Python 3.5 Paul Rubin <no.email@nospam.invalid> - 2016-02-20 23:34 -0800
        Re: How the heck does async/await work in Python 3.5 Ian Kelly <ian.g.kelly@gmail.com> - 2016-02-20 00:48 -0700
        Re: How the heck does async/await work in Python 3.5 Chris Angelico <rosuav@gmail.com> - 2016-02-20 18:57 +1100
        Re: How the heck does async/await work in Python 3.5 Ian Kelly <ian.g.kelly@gmail.com> - 2016-02-20 01:14 -0700
        Re: How the heck does async/await work in Python 3.5 Chris Angelico <rosuav@gmail.com> - 2016-02-20 19:49 +1100
        Re: How the heck does async/await work in Python 3.5 Ian Kelly <ian.g.kelly@gmail.com> - 2016-02-20 02:11 -0700
        Re: How the heck does async/await work in Python 3.5 Ian Kelly <ian.g.kelly@gmail.com> - 2016-02-20 02:21 -0700
      Re: How the heck does async/await work in Python 3.5 Christian Gollwitzer <auriocus@gmx.de> - 2016-02-20 07:53 +0100
        Re: How the heck does async/await work in Python 3.5 "Sven R. Kunze" <srkunze@mail.de> - 2016-02-22 23:16 +0100
        Re: How the heck does async/await work in Python 3.5 Ian Kelly <ian.g.kelly@gmail.com> - 2016-02-22 17:48 -0700
        Re: How the heck does async/await work in Python 3.5 "Sven R. Kunze" <srkunze@mail.de> - 2016-02-23 17:50 +0100
        Re: How the heck does async/await work in Python 3.5 Ian Kelly <ian.g.kelly@gmail.com> - 2016-02-23 10:37 -0700
        Re: How the heck does async/await work in Python 3.5 "Sven R. Kunze" <srkunze@mail.de> - 2016-02-23 20:42 +0100
        Re: How the heck does async/await work in Python 3.5 "Sven R. Kunze" <srkunze@mail.de> - 2016-02-23 22:05 +0100
        Re: [Python-ideas] How the heck does async/await work in Python 3.5 "Joao S. O. Bueno" <jsbueno@python.org.br> - 2016-02-23 18:25 -0300
        Re: [Python-ideas] How the heck does async/await work in Python 3.5 Paul Moore <p.f.moore@gmail.com> - 2016-02-24 09:59 +0000
        Re: [Python-ideas] How the heck does async/await work in Python 3.5 Victor Stinner <victor.stinner@gmail.com> - 2016-02-24 11:01 +0100
        Re: [Python-ideas] How the heck does async/await work in Python 3.5 王珺 <wjun77@gmail.com> - 2016-02-24 18:40 +0800
          Re: [Python-ideas] How the heck does async/await work in Python 3.5 Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2016-02-25 10:00 +1300
            Re: [Python-ideas] How the heck does async/await work in Python 3.5 王珺 <wjun77@gmail.com> - 2016-02-25 08:40 +0800
            Re: [Python-ideas] How the heck does async/await work in Python 3.5 Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2016-02-24 20:37 -0500
        Re: [Python-ideas] How the heck does async/await work in Python 3.5 Tem Pl <rtempl31@gmail.com> - 2016-02-24 06:39 -0800
          Re: [Python-ideas] How the heck does async/await work in Python 3.5 Marko Rauhamaa <marko@pacujo.net> - 2016-02-24 17:23 +0200
            Re: [Python-ideas] How the heck does async/await work in Python 3.5 Ian Kelly <ian.g.kelly@gmail.com> - 2016-02-24 08:41 -0700
              Re: [Python-ideas] How the heck does async/await work in Python 3.5 Marko Rauhamaa <marko@pacujo.net> - 2016-02-24 18:13 +0200
                Re: [Python-ideas] How the heck does async/await work in Python 3.5 Ian Kelly <ian.g.kelly@gmail.com> - 2016-02-24 09:47 -0700

Page 2 of 2 — ← Prev page 1 [2]

#103437 — Re: [Python-ideas] How the heck does async/await work in Python 3.5

Today I also stumbled on this helpful "essay" from Brett Cannon about
the same subject

http://www.snarky.ca/how-the-heck-does-async-await-work-in-python-3-5

On 23 February 2016 at 18:05, Sven R. Kunze <srkunze@mail.de> wrote:
> On 20.02.2016 07:53, Christian Gollwitzer wrote:
>
> If you have difficulties wit hthe overall concept, and if you are open to
> discussions in another language, take a look at this video:
>
> https://channel9.msdn.com/Shows/C9-GoingNative/GoingNative-39-await-co-routines
>
> MS has added coroutine support with very similar syntax to VC++ recently,
> and the developer tries to explain it to the "stackful" programmers.
>
>
> Because of this thread, I finally finished an older post collecting valuable
> insights from last year discussions regarding concurrency modules available
> in Python: http://srkunze.blogspot.com/2016/02/concurrency-in-python.html It
> appears to me that it would fit here well.
>
> @python-ideas
> Back then, the old thread ("Concurrency Modules") was like basically meant
> to result in something useful. I hope the post covers the essence of the
> discussion.
> Some even suggested putting the table into the Python docs. I am unaware of
> the formal procedure here but I would be glad if somebody could point be at
> the right direction if that the survey table is wanted in the docs.
>
> Best,
> Sven
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

[toc] | [prev] | [next] | [standalone]

#103438 — Re: [Python-ideas] How the heck does async/await work in Python 3.5

On 24 February 2016 at 02:37, Terry Reedy <tjreedy@udel.edu> wrote:
>
> In this essay, Brett says that asyncio added an event loop to Python. It
> did, but it was the second.  The tk event loop was added about 20 years ago
> with tkinter.

One of the things I would love to see (but don't have the time to work
on) is a GUI event loop based around async/await. It would be a very
useful example to make it clear to people that async/await isn't just
about network protocols.

Paul

[toc] | [prev] | [next] | [standalone]

#103439 — Re: [Python-ideas] How the heck does async/await work in Python 3.5

See also Doug Hellmann article on asyncio, from its serie of "Python 3
Module of the Week" articles:
https://pymotw.com/3/asyncio/index.html

Victor

2016-02-23 22:25 GMT+01:00 Joao S. O. Bueno <jsbueno@python.org.br>:
> Today I also stumbled on this helpful "essay" from Brett Cannon about
> the same subject
>
> http://www.snarky.ca/how-the-heck-does-async-await-work-in-python-3-5
>
> On 23 February 2016 at 18:05, Sven R. Kunze <srkunze@mail.de> wrote:
>> On 20.02.2016 07:53, Christian Gollwitzer wrote:
>>
>> If you have difficulties wit hthe overall concept, and if you are open to
>> discussions in another language, take a look at this video:
>>
>> https://channel9.msdn.com/Shows/C9-GoingNative/GoingNative-39-await-co-routines
>>
>> MS has added coroutine support with very similar syntax to VC++ recently,
>> and the developer tries to explain it to the "stackful" programmers.
>>
>>
>> Because of this thread, I finally finished an older post collecting valuable
>> insights from last year discussions regarding concurrency modules available
>> in Python: http://srkunze.blogspot.com/2016/02/concurrency-in-python.html It
>> appears to me that it would fit here well.
>>
>> @python-ideas
>> Back then, the old thread ("Concurrency Modules") was like basically meant
>> to result in something useful. I hope the post covers the essence of the
>> discussion.
>> Some even suggested putting the table into the Python docs. I am unaware of
>> the formal procedure here but I would be glad if somebody could point be at
>> the right direction if that the survey table is wanted in the docs.
>>
>> Best,
>> Sven
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

[toc] | [prev] | [next] | [standalone]

#103440 — Re: [Python-ideas] How the heck does async/await work in Python 3.5

It seems an event loop is required for all async programs in python, but
sometimes I need only lazy i/o. Is it possible with asyncio?


Suppose io_operation() takes 3 seconds, then how can I write something like

future = io_operation()
print('Start')
time.sleep(1)
print('Something')
time.sleep(2)
print(future.result())

that print 'Start' immediately and the result of io_operation() 3 seconds
later. The actual io operations are done by the io device that the main
(and only) thread is not interrupted at all, if my understanding is
correct.

2016-02-24 5:05 GMT+08:00 Sven R. Kunze <srkunze@mail.de>:

> On 20.02.2016 07:53, Christian Gollwitzer wrote:
>
> If you have difficulties wit hthe overall concept, and if you are open to
> discussions in another language, take a look at this video:
>
>
> https://channel9.msdn.com/Shows/C9-GoingNative/GoingNative-39-await-co-routines
>
> MS has added coroutine support with very similar syntax to VC++ recently,
> and the developer tries to explain it to the "stackful" programmers.
>
>
> Because of this thread, I finally finished an older post collecting
> valuable insights from last year discussions regarding concurrency modules
> available in Python:
> http://srkunze.blogspot.com/2016/02/concurrency-in-python.html It appears
> to me that it would fit here well.
>
> @python-ideas
> Back then, the old thread ("Concurrency Modules") was like basically meant
> to result in something useful. I hope the post covers the essence of the
> discussion.
> Some even suggested putting the table into the Python docs. I am unaware
> of the formal procedure here but I would be glad if somebody could point be
> at the right direction if that the survey table is wanted in the docs.
>
> Best,
> Sven
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>

[toc] | [prev] | [next] | [standalone]

#103460 — Re: [Python-ideas] How the heck does async/await work in Python 3.5

王珺 wrote:
> Suppose io_operation() takes 3 seconds, then how can I write something like
> 
> future = io_operation()
> print('Start')
> time.sleep(1)
> print('Something')
> time.sleep(2)
> print(future.result())
> 
> that print 'Start' immediately and the result of io_operation() 3 seconds
> later.

Yes, Python can do this, but you probably need to use real
threads. The only exception would be if io_operation() were
something you could fire off with a single system call that's
guaranteed not to block, and then wait for the result later.
Opportunities for things like that are rare in unix.
(Windows might be different, I'm not sure.)

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#103468 — Re: [Python-ideas] How the heck does async/await work in Python 3.5

I think this is possible if I understand what happens under the hood.
I wonder how event loop and async io functions such as
asyncio.open_connection cooperate to do async io in one thread. Maybe it
exploits low-level details and is OS or even device specific. I think I
should take a look at the source code when I have time.

2016-02-25 5:00 GMT+08:00 Gregory Ewing <greg.ewing@canterbury.ac.nz>:

> 王珺 wrote:
>
>> Suppose io_operation() takes 3 seconds, then how can I write something
>> like
>>
>> future = io_operation()
>> print('Start')
>> time.sleep(1)
>> print('Something')
>> time.sleep(2)
>> print(future.result())
>>
>> that print 'Start' immediately and the result of io_operation() 3 seconds
>> later.
>>
>
> Yes, Python can do this, but you probably need to use real
> threads. The only exception would be if io_operation() were
> something you could fire off with a single system call that's
> guaranteed not to block, and then wait for the result later.
> Opportunities for things like that are rare in unix.
> (Windows might be different, I'm not sure.)
>
> --
> Greg
> --
> https://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

#103475 — Re: [Python-ideas] How the heck does async/await work in Python 3.5

On Thu, 25 Feb 2016 10:00:29 +1300, Gregory Ewing
<greg.ewing@canterbury.ac.nz> declaimed the following:

>?? wrote:
>> Suppose io_operation() takes 3 seconds, then how can I write something like
>> 
>> future = io_operation()
>> print('Start')
>> time.sleep(1)
>> print('Something')
>> time.sleep(2)
>> print(future.result())
>> 
>> that print 'Start' immediately and the result of io_operation() 3 seconds
>> later.
>
>Yes, Python can do this, but you probably need to use real
>threads. The only exception would be if io_operation() were
>something you could fire off with a single system call that's
>guaranteed not to block, and then wait for the result later.
>Opportunities for things like that are rare in unix.
>(Windows might be different, I'm not sure.)
>
	But was common under VMS
https://en.wikipedia.org/wiki/QIO

	I believe Windows is supposed to be similar, but QIO status words and
event flags were understandable to me (and close match to AmigaOS too <G>);
Windows somehow treats the I/O channel itself as the waitable object (well,
looks like one embeds it into the I/O call -- but has to create the channel
in "overlapped" mode where VMS differentiated on the I/O call itself: QIO
vs QIOW; Amiga differentiated with DoIO and SendIO)
https://msdn.microsoft.com/en-us/library/windows/desktop/ms686358%28v=vs.85%29.aspx

http://wiki.amigaos.net/wiki/Exec_Device_I/O#Synchronous_vs._Asynchronous_Requests
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
    wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#103445 — Re: [Python-ideas] How the heck does async/await work in Python 3.5

Here are some concurrency benchmarks for python vs other languages. 

https://github.com/atemerev/skynet/pull/53

Is there something wrong with this implementation?

"Hope I suck at coroutines, because the results are abysmal.

I get around 63s on my i5 MacBook Air Early 2015. For reference, the go 
version got around 700ms with GOMAXPROCS=4 and 1.7s with GOMAXPROCS=1 on 
this same machine."

On Wednesday, February 24, 2016 at 5:02:16 AM UTC-5, Victor Stinner wrote:
>
> See also Doug Hellmann article on asyncio, from its serie of "Python 3 
> Module of the Week" articles: 
> https://pymotw.com/3/asyncio/index.html 
>
> Victor 
>
> 2016-02-23 22:25 GMT+01:00 Joao S. O. Bueno <jsb...@python.org.br 
> <javascript:>>: 
> > Today I also stumbled on this helpful "essay" from Brett Cannon about 
> > the same subject 
> > 
> > http://www.snarky.ca/how-the-heck-does-async-await-work-in-python-3-5 
> > 
> > On 23 February 2016 at 18:05, Sven R. Kunze <srk...@mail.de 
> <javascript:>> wrote: 
> >> On 20.02.2016 07:53, Christian Gollwitzer wrote: 
> >> 
> >> If you have difficulties wit hthe overall concept, and if you are open 
> to 
> >> discussions in another language, take a look at this video: 
> >> 
> >> 
> https://channel9.msdn.com/Shows/C9-GoingNative/GoingNative-39-await-co-routines 
> >> 
> >> MS has added coroutine support with very similar syntax to VC++ 
> recently, 
> >> and the developer tries to explain it to the "stackful" programmers. 
> >> 
> >> 
> >> Because of this thread, I finally finished an older post collecting 
> valuable 
> >> insights from last year discussions regarding concurrency modules 
> available 
> >> in Python: 
> http://srkunze.blogspot.com/2016/02/concurrency-in-python.html It 
> >> appears to me that it would fit here well. 
> >> 
> >> @python-ideas 
> >> Back then, the old thread ("Concurrency Modules") was like basically 
> meant 
> >> to result in something useful. I hope the post covers the essence of 
> the 
> >> discussion. 
> >> Some even suggested putting the table into the Python docs. I am 
> unaware of 
> >> the formal procedure here but I would be glad if somebody could point 
> be at 
> >> the right direction if that the survey table is wanted in the docs. 
> >> 
> >> Best, 
> >> Sven 
> >> 
> >> _______________________________________________ 
> >> Python-ideas mailing list 
> >> Python...@python.org <javascript:> 
> >> https://mail.python.org/mailman/listinfo/python-ideas 
> >> Code of Conduct: http://python.org/psf/codeofconduct/ 
> > _______________________________________________ 
> > Python-ideas mailing list 
> > Python...@python.org <javascript:> 
> > https://mail.python.org/mailman/listinfo/python-ideas 
> > Code of Conduct: http://python.org/psf/codeofconduct/ 
> _______________________________________________ 
> Python-ideas mailing list 
> Python...@python.org <javascript:> 
> https://mail.python.org/mailman/listinfo/python-ideas 
> Code of Conduct: http://python.org/psf/codeofconduct/ 
>

[toc] | [prev] | [next] | [standalone]

#103447 — Re: [Python-ideas] How the heck does async/await work in Python 3.5

Tem Pl <rtempl31@gmail.com>:

> Here are some concurrency benchmarks for python vs other languages. 
>
> https://github.com/atemerev/skynet/pull/53
>
> Is there something wrong with this implementation?

It's a "fork bomb".


Marko

[toc] | [prev] | [next] | [standalone]

#103448 — Re: [Python-ideas] How the heck does async/await work in Python 3.5

On Wed, Feb 24, 2016 at 8:23 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Tem Pl <rtempl31@gmail.com>:
>
>> Here are some concurrency benchmarks for python vs other languages.
>>
>> https://github.com/atemerev/skynet/pull/53
>>
>> Is there something wrong with this implementation?
>
> It's a "fork bomb".

Isn't that the point of the benchmark?

But yeah, I tried playing with this and here are my results. It's
clearly pushing all those coroutines down into the event loop that
causes the problem. If you remove the asyncio.as_completed calls and
just await the descendant coroutines in order, then it's a lot faster
-- but this may be technically cheating because then the coroutines
are effectively just running recursively without involving the event
loop, so there's no exercise of actual concurrency.

# Straight-up recursion: 400 ms
def func(level=0, index=0):
    if level < LEVELS:
        sons = [func(level=level+1, index=index*SONS + x) for x in range(SONS)]
        sum_ = 0
        for son in sons:
            sum_ += son
        return sum_
    else:
        return index

import timeit
print(timeit.repeat('func()', 'from __main__ import func', number=10))
# [4.016848850995302, 4.1330014310078695, 4.149791953997919]

# Concurrent coroutines with event loop: 30 s
import asyncio
async def coroutine(level=0, index=0):
    if level < LEVELS:
        sons = [coroutine(level=level+1, index=index*SONS + x) for x
in range(SONS)]
        sum_ = 0
        for f in asyncio.as_completed(sons):
            got = await f
            sum_ += got
        return sum_
    else:
        return index

print(timeit.repeat('loop.run_until_complete(coroutine())', 'from
__main__ import coroutine; import asyncio; loop =
asyncio.get_event_loop()', number=1))
# [29.884846250002738, 30.26590966898948, 30.716448744002264]

# Recursion with coroutines: 600 ms
async def coro2(level=0, index=0):
    if level < LEVELS:
        sons = [coro2(level=level+1, index=index*SONS + x) for x in range(SONS)]
        sum_ = 0
        for son in sons:
            got = await son
            sum_ += got
        return sum_
    else:
        return index

print(timeit.repeat('loop.run_until_complete(coro2())', 'from __main__
import coro2; import asyncio; loop = asyncio.get_event_loop()',
number=1))
# [0.6264017040084582, 0.590631059021689, 0.5807875629980117]

# Recursive coroutines, no event loop: 600 ms
print(timeit.repeat('list(coro2().__await__())', 'from __main__ import
coro2', number=10))
# [6.028770218021236, 6.002665672975127, 5.987304503010819]

[toc] | [prev] | [next] | [standalone]

#103453 — Re: [Python-ideas] How the heck does async/await work in Python 3.5

Ian Kelly <ian.g.kelly@gmail.com>:

> On Wed, Feb 24, 2016 at 8:23 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
>> Tem Pl <rtempl31@gmail.com>:
>>> Is there something wrong with this implementation?
>>
>> It's a "fork bomb".
>
> Isn't that the point of the benchmark?

I don't quite see the point of the program as it doesn't resemble
anything I'd ever have an urge to write.

As for benchmarks, Python is not supposed to be fast nor small. Python
is actually quite slow and the objects have a large footprint. However,
Python's strength is in its powerful expressivity.

Still, it may be that coroutines end up being especially slow; that's
how they played out in my head when I first was imagining what all those
nested generators must be doing to produce their magic. Also, I'm afraid
the expressivity part suffers a blow with the whole coroutine paradigm
-- without having the otherworldly appeal of Scheme's continuations.


Marko

[toc] | [prev] | [next] | [standalone]

#103454 — Re: [Python-ideas] How the heck does async/await work in Python 3.5

On Wed, Feb 24, 2016 at 9:13 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Ian Kelly <ian.g.kelly@gmail.com>:
>
>> On Wed, Feb 24, 2016 at 8:23 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
>>> Tem Pl <rtempl31@gmail.com>:
>>>> Is there something wrong with this implementation?
>>>
>>> It's a "fork bomb".
>>
>> Isn't that the point of the benchmark?
>
> I don't quite see the point of the program as it doesn't resemble
> anything I'd ever have an urge to write.

That's reasonable. asyncio is designed for I/O-bound work, so a fair
response to the benchmark is that subjecting it to a CPU-bound stress
test is perhaps missing the point.

I was curious about how much the selector-based event loop might be
influencing the timing, so I tried instrumenting the selector.

class ProfiledSelector(selectors.DefaultSelector):
    def __init__(self):
        super().__init__()
        self._calls = collections.Counter()
    def select(self, timeout=None):
        self._calls['select'] += 1
        return super().select(timeout)
    def register(self, fileobj, events, data=None):
        self._calls['register'] += 1
        return super().register(fileobj, events, data)

selector = ProfiledSelector()
loop = asyncio.SelectorEventLoop(selector)
asyncio.set_event_loop(loop)
loop.run_until_complete(coroutine())
# 499999500000
selector._calls
# Counter({'select': 20, 'register': 1})

Only 20 iterations of the event loop? That doesn't seem unreasonable.

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.python

csiph-web