Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #103265

Re: asyncio - run coroutine in the background

Show key headers only | View raw

> getaddrinfo is a notorious pain but I think it's just a library issue; an
async version should be possible in principle.  How does Twisted handle
it?  Does it have a version?

I think we're a little outside the scope of OP's question at this point,
but for the sake of answering this:

There are a few cases that I know of where Twisted uses the standard lib
socket DNS methods. One is when resolving names to IPv6 addresses [1] when
creating a client connection to a remote source. The other is in the
default DNS resolver that is installed in the reactor [2]. Creating client
connections allows the call to 'getaddrinfo' to block without mitigation.
The default DNS resolver, unfortunately, dispatches calls of
'gethostbyname' to a thread pool.

Without seeing the commit history, I'd assume the use of 'socket' and
threads by default is an artifact that predates the implementation of the
DNS protocol in Twisted. Twisted has, in 'twisted.names' [3], a DNS
protocol that uses UDP and leverages the reactor appropriately. Thankfully,
Twisted has a reactor method called 'installResolver' [4] that allows you
to hook in any DNS resolver implementation you want so you aren't stuck
using the default, threaded implementation.

As far as asyncio, it also defaults to an implementation that delegates to
an executor (default: threadpool). Unlike Twisted, though, it appears to
require a subclass of the event loop to override the 'getaddrinfo' method
[5].

[1]
https://github.com/twisted/twisted/blob/trunk/twisted/internet/tcp.py#L622
[2]
https://github.com/twisted/twisted/blob/trunk/twisted/internet/base.py#L257
[3] https://github.com/twisted/twisted/tree/trunk/twisted/names
[4]
https://github.com/twisted/twisted/blob/trunk/twisted/internet/base.py#L509
[5]
https://github.com/python/cpython/blob/master/Lib/asyncio/base_events.py#L572

On Sat, Feb 20, 2016, 03:31 Marko Rauhamaa <marko@pacujo.net> wrote:

> Paul Rubin <no.email@nospam.invalid>:
>
> > I've just felt depressed whenever I've looked at any Python async
> > stuff. I've written many Python programs with threads and not gotten
> > into the trouble that people keep warning about.
>
> Programming-model-wise, asyncio is virtually identical with threads. In
> each, I dislike the implicit state concept. I want the state to stand
> out with big block letters.
>
> > I've just felt depressed whenever I've looked at any Python async
> > stuff. I've written many Python programs with threads and not gotten
> > into the trouble that people keep warning about. But I haven't really
> > understood the warnings, so maybe they know something I don't. I just
> > write in a multiprocessing style, with every mutable object owned by
> > exactly one thread and accessed only by RPC through queues, sort of a
> > poor man's Erlang. There's a performance hit but there's a much bigger
> > one from using Python in the first place, so I just live with it.
>
> Good for you if you have been able to choose your own programming model.
> Most people have to deal with a legacy mess. Also, maintainers who
> inherit your tidy code might not be careful to ship only nonmutable
> objects in the queues.
>
> Your way of using threads works, of course, with the caveat that it is
> not possible to get rid of a blocking thread from the outside. With
> asyncio, you can at least cancel tasks.
>
>
> Marko
> --
> https://mail.python.org/mailman/listinfo/python-list
>

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-15 08:35 +0200
  Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-15 08:54 +0200
    Re: asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-15 09:16 +0200
      Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-15 09:34 +0200
      Re: asyncio - run coroutine in the background Paul Rubin <no.email@nospam.invalid> - 2016-02-14 23:39 -0800
        Re: asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-15 10:17 +0200
        Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-15 13:05 +0200
        Re: asyncio - run coroutine in the background Chris Angelico <rosuav@gmail.com> - 2016-02-16 16:51 +1100
        Re: asyncio - run coroutine in the background Kevin Conway <kevinjacobconway@gmail.com> - 2016-02-16 13:22 +0000
          Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 16:17 +0200
            Re: asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-16 16:36 +0200
            Re: asyncio - run coroutine in the background Kevin Conway <kevinjacobconway@gmail.com> - 2016-02-16 14:54 +0000
            Re: asyncio - run coroutine in the background Steven D'Aprano <steve@pearwood.info> - 2016-02-17 02:17 +1100
              Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 19:12 +0200
                Re: asyncio - run coroutine in the background Paul Rubin <no.email@nospam.invalid> - 2016-02-17 20:38 -0800
                Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-18 08:10 +0200
              Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 19:13 +0200
              Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 19:14 +0200
              Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 19:15 +0200
                Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 19:15 +0200
                Re: asyncio - run coroutine in the background Robin Becker <robin@reportlab.com> - 2016-02-16 17:52 +0000
            Re: asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-16 17:21 +0200
              Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 19:20 +0200
                Re: asyncio - run coroutine in the background Paul Rubin <no.email@nospam.invalid> - 2016-02-19 23:40 -0800
                Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-20 10:13 +0200
                Re: asyncio - run coroutine in the background Paul Rubin <no.email@nospam.invalid> - 2016-02-20 00:37 -0800
                Re: asyncio - run coroutine in the background Chris Angelico <rosuav@gmail.com> - 2016-02-20 19:52 +1100
                Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-20 10:59 +0200
                Re: asyncio - run coroutine in the background Chris Angelico <rosuav@gmail.com> - 2016-02-20 20:02 +1100
                Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-20 11:28 +0200
                Re: asyncio - run coroutine in the background Kevin Conway <kevinjacobconway@gmail.com> - 2016-02-20 13:52 +0000
                Re: asyncio - run coroutine in the background "Martin A. Brown" <martin@linux-ip.net> - 2016-02-20 09:45 -0800
                Re: asyncio - run coroutine in the background Chris Angelico <rosuav@gmail.com> - 2016-02-21 08:47 +1100
            Re: asyncio - run coroutine in the background Chris Angelico <rosuav@gmail.com> - 2016-02-17 02:28 +1100
            Re: asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-16 17:45 +0200
        Re: asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-16 15:52 +0200

csiph-web