Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #102941 > unrolled thread

asyncio - run coroutine in the background

Back to article view | Back to comp.lang.python

Contents

  asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-15 08:35 +0200
    Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-15 08:54 +0200
      Re: asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-15 09:16 +0200
        Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-15 09:34 +0200
        Re: asyncio - run coroutine in the background Paul Rubin <no.email@nospam.invalid> - 2016-02-14 23:39 -0800
          Re: asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-15 10:17 +0200
          Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-15 13:05 +0200
          Re: asyncio - run coroutine in the background Chris Angelico <rosuav@gmail.com> - 2016-02-16 16:51 +1100
          Re: asyncio - run coroutine in the background Kevin Conway <kevinjacobconway@gmail.com> - 2016-02-16 13:22 +0000
            Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 16:17 +0200
              Re: asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-16 16:36 +0200
              Re: asyncio - run coroutine in the background Kevin Conway <kevinjacobconway@gmail.com> - 2016-02-16 14:54 +0000
              Re: asyncio - run coroutine in the background Steven D'Aprano <steve@pearwood.info> - 2016-02-17 02:17 +1100
                Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 19:12 +0200
                  Re: asyncio - run coroutine in the background Paul Rubin <no.email@nospam.invalid> - 2016-02-17 20:38 -0800
                    Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-18 08:10 +0200
                Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 19:13 +0200
                Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 19:14 +0200
                Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 19:15 +0200
                  Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 19:15 +0200
                    Re: asyncio - run coroutine in the background Robin Becker <robin@reportlab.com> - 2016-02-16 17:52 +0000
              Re: asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-16 17:21 +0200
                Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 19:20 +0200
                  Re: asyncio - run coroutine in the background Paul Rubin <no.email@nospam.invalid> - 2016-02-19 23:40 -0800
                    Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-20 10:13 +0200
                      Re: asyncio - run coroutine in the background Paul Rubin <no.email@nospam.invalid> - 2016-02-20 00:37 -0800
                        Re: asyncio - run coroutine in the background Chris Angelico <rosuav@gmail.com> - 2016-02-20 19:52 +1100
                          Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-20 10:59 +0200
                            Re: asyncio - run coroutine in the background Chris Angelico <rosuav@gmail.com> - 2016-02-20 20:02 +1100
                        Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-20 11:28 +0200
                          Re: asyncio - run coroutine in the background Kevin Conway <kevinjacobconway@gmail.com> - 2016-02-20 13:52 +0000
                        Re: asyncio - run coroutine in the background "Martin A. Brown" <martin@linux-ip.net> - 2016-02-20 09:45 -0800
                        Re: asyncio - run coroutine in the background Chris Angelico <rosuav@gmail.com> - 2016-02-21 08:47 +1100
              Re: asyncio - run coroutine in the background Chris Angelico <rosuav@gmail.com> - 2016-02-17 02:28 +1100
              Re: asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-16 17:45 +0200
          Re: asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-16 15:52 +0200

Page 2 of 2 — ← Prev page 1 [2]

#103025

On 16/02/2016 17:15, Marko Rauhamaa wrote:
> Marko Rauhamaa <marko@pacujo.net>:
>
>> Sure:
>
> Sorry for the multiple copies.
>
>
> Marko
>

I thought perhaps background jobs were sending them :)
-- 
Robin Becker

[toc] | [prev] | [next] | [standalone]

#103014

"Kevin Conway"  wrote in message 
news:CAKF=+dhXZ=yax8STAWr_gjX3Tg8yUjPRJG-7yM2_bRV2Kxm3Jg@mail.gmail.com...
>
> > My background task does take a long time to run - about 10 seconds - but
> > most of that time is spent waiting for database responses, which is 
> > handled
> > in another thread.
>
> Something else to look into is an asyncio driver for your database
> connections. Threads aren't inherently harmful, but using them to achieve
> async networking when running asyncio is a definite code smell since that
> is precisely the problem asyncio is supposed to solve for.
>

Maybe I have not explained very well. I am not using threads to achieve 
async networking. I am using asyncio in a client server environment, and it 
works very well. If a client request involves a database query, I use a 
thread to perform that so that it does not slow down the other users. I 
usually want the originating client to block until I have a response, so I 
use 'await'. However, occasionally the request takes some time, and it is 
not necessary for the client to wait for the response, so I want to unblock 
the client straight away, run the task in the background, and then notify 
the client when the task is complete. This is where your suggestion of 
'ensure_future' does the job perfectly.

I would love to drive the database asynchronously, but of the three 
databases I use, only psycopg2 seems to have asyncio support. As my 
home-grown solution (using queues) seems to be working well so far, I am 
sticking with that until I start to experience responsiveness issues. If 
that happens, my first line of attack will be to switch from threads to 
processes.

Hope this makes sense.

Frank

[toc] | [prev] | [next] | [standalone]

#103023

"Frank Millman" <frank@chagford.com>:

> I would love to drive the database asynchronously, but of the three
> databases I use, only psycopg2 seems to have asyncio support.

Yes, asyncio is at its infancy. There needs to be a moratorium on
blocking I/O.


Marko

[toc] | [prev] | [next] | [standalone]

#103241

Marko Rauhamaa <marko@pacujo.net> writes:
> "Frank Millman" <frank@chagford.com>:
>> I would love to drive the database asynchronously, but of the three
>> databases I use, only psycopg2 seems to have asyncio support.
> Yes, asyncio is at its infancy. There needs to be a moratorium on
> blocking I/O.

Unfortunately there appears to be no way to open a file in Linux without
at least potentially blocking (slow disk or whatever).  You need
separate threads or processes to do the right thing.

[toc] | [prev] | [next] | [standalone]

#103248

Paul Rubin <no.email@nospam.invalid>:

> Marko Rauhamaa <marko@pacujo.net> writes:
>> "Frank Millman" <frank@chagford.com>:
>>> I would love to drive the database asynchronously, but of the three
>>> databases I use, only psycopg2 seems to have asyncio support.
>> Yes, asyncio is at its infancy. There needs to be a moratorium on
>> blocking I/O.
>
> Unfortunately there appears to be no way to open a file in Linux
> without at least potentially blocking (slow disk or whatever). You
> need separate threads or processes to do the right thing.

I have been wondering about the same thing. It would appear that disk
I/O is considered nonblocking at a very deep level:

 * O_NONBLOCK doesn't have an effect

 * a process waiting for the disk to respond cannot receive a signal

 * a process waiting for the disk to respond stays in the "ready" state

Note that

 * most disk I/O operates on a RAM cache that is flushed irregularly

 * memory mapping and swapping make disk I/O and RAM access two sides of
   the same coin

 * page faults can turn any assembly language instruction into a
   blocking disk I/O operation

 * ordinary disks don't provide for much parallelism; processes are
   usually serialized for disk I/O

If the file system happens to be NFS, a networking issue may paralyze
the whole system.

...

On the networking side, there is also a dangerous blocking operation:
socket.getaddrinfo() (and friends). As a consequence, socket.bind(),
socket.connect() may block indefinitely. In fact, even asyncio's
BaseEventLoop.create_server() and BaseEventLoop.create_sonnection() may
block indefinitely without yielding.

SEE ALSO
    getaddrinfo_a(3)


Marko

[toc] | [prev] | [next] | [standalone]

#103251

Marko Rauhamaa <marko@pacujo.net> writes:
> It would appear that disk I/O is considered nonblocking at a very deep
> level:
>  * O_NONBLOCK doesn't have an effect
>  * a process waiting for the disk to respond cannot receive a signal
>  * a process waiting for the disk to respond stays in the "ready" state

You can handle those issues with AIO.  It's open(2) that seems to have
no asynchronous analog as far as I can tell.  Actually, looking at the
AIO man pages, it appears that the Linux kernel currently doesn't
support it and it's instead simulated by a userspace library using
threads.  I didn't realize that before.  But AIO is at least specified
by POSIX, and there was some kernel work (io_setup(2) etc.) that may or
may not still be in progress.  It also doesn't have an open(2) analog,
sigh.

> On the networking side, there is also a dangerous blocking operation:
> socket.getaddrinfo() (and friends). As a consequence, socket.bind(),
> socket.connect() may block indefinitely. In fact, even asyncio's
> BaseEventLoop.create_server() and BaseEventLoop.create_sonnection() may
> block indefinitely without yielding.

getaddrinfo is a notorious pain but I think it's just a library issue;
an async version should be possible in principle.  How does Twisted
handle it?  Does it have a version?  

I've just felt depressed whenever I've looked at any Python async stuff.
I've written many Python programs with threads and not gotten into the
trouble that people keep warning about.  But I haven't really understood
the warnings, so maybe they know something I don't.  I just write in a
multiprocessing style, with every mutable object owned by exactly one
thread and accessed only by RPC through queues, sort of a poor man's
Erlang.  There's a performance hit but there's a much bigger one from
using Python in the first place, so I just live with it.

[toc] | [prev] | [next] | [standalone]

#103256

On Sat, Feb 20, 2016 at 7:37 PM, Paul Rubin <no.email@nospam.invalid> wrote:
> getaddrinfo is a notorious pain but I think it's just a library issue;
> an async version should be possible in principle.  How does Twisted
> handle it?  Does it have a version?

In a (non-Python) program of mine, I got annoyed by synchronous name
lookups, so I hacked around it: instead of using the regular library
functions, I just do a DNS lookup directly (which can then be
event-based - send a UDP packet, get notified when a UDP packet
arrives). Downside: Ignores /etc/nsswitch.conf and /etc/hosts, and
goes straight to the name server. Upside: Is able to do its own
caching, since the DNS library gives me the TTLs, but
gethostbyname/getaddrinfo won't.

ChrisA

[toc] | [prev] | [next] | [standalone]

#103257

Chris Angelico <rosuav@gmail.com>:

> In a (non-Python) program of mine, I got annoyed by synchronous name
> lookups, so I hacked around it: instead of using the regular library
> functions, I just do a DNS lookup directly (which can then be
> event-based - send a UDP packet, get notified when a UDP packet
> arrives). Downside: Ignores /etc/nsswitch.conf and /etc/hosts, and
> goes straight to the name server. Upside: Is able to do its own
> caching, since the DNS library gives me the TTLs, but
> gethostbyname/getaddrinfo won't.

Ditto in a Python program of mine, although I don't bother with caching:
the DNS server is perfectly capable of caching the entries for me.


Marko

[toc] | [prev] | [next] | [standalone]

#103258

On Sat, Feb 20, 2016 at 7:59 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Chris Angelico <rosuav@gmail.com>:
>
>> In a (non-Python) program of mine, I got annoyed by synchronous name
>> lookups, so I hacked around it: instead of using the regular library
>> functions, I just do a DNS lookup directly (which can then be
>> event-based - send a UDP packet, get notified when a UDP packet
>> arrives). Downside: Ignores /etc/nsswitch.conf and /etc/hosts, and
>> goes straight to the name server. Upside: Is able to do its own
>> caching, since the DNS library gives me the TTLs, but
>> gethostbyname/getaddrinfo won't.
>
> Ditto in a Python program of mine, although I don't bother with caching:
> the DNS server is perfectly capable of caching the entries for me.

If you know you have a local DNS server, sure. Mine is written for a
generic situation where that can't be depended on, so it caches
itself. But it's no big deal.

ChrisA

[toc] | [prev] | [next] | [standalone]

#103262

Paul Rubin <no.email@nospam.invalid>:

> I've just felt depressed whenever I've looked at any Python async
> stuff. I've written many Python programs with threads and not gotten
> into the trouble that people keep warning about.

Programming-model-wise, asyncio is virtually identical with threads. In
each, I dislike the implicit state concept. I want the state to stand
out with big block letters.

> I've just felt depressed whenever I've looked at any Python async
> stuff. I've written many Python programs with threads and not gotten
> into the trouble that people keep warning about. But I haven't really
> understood the warnings, so maybe they know something I don't. I just
> write in a multiprocessing style, with every mutable object owned by
> exactly one thread and accessed only by RPC through queues, sort of a
> poor man's Erlang. There's a performance hit but there's a much bigger
> one from using Python in the first place, so I just live with it.

Good for you if you have been able to choose your own programming model.
Most people have to deal with a legacy mess. Also, maintainers who
inherit your tidy code might not be careful to ship only nonmutable
objects in the queues.

Your way of using threads works, of course, with the caveat that it is
not possible to get rid of a blocking thread from the outside. With
asyncio, you can at least cancel tasks.


Marko

[toc] | [prev] | [next] | [standalone]

#103265

> getaddrinfo is a notorious pain but I think it's just a library issue; an
async version should be possible in principle.  How does Twisted handle
it?  Does it have a version?

I think we're a little outside the scope of OP's question at this point,
but for the sake of answering this:

There are a few cases that I know of where Twisted uses the standard lib
socket DNS methods. One is when resolving names to IPv6 addresses [1] when
creating a client connection to a remote source. The other is in the
default DNS resolver that is installed in the reactor [2]. Creating client
connections allows the call to 'getaddrinfo' to block without mitigation.
The default DNS resolver, unfortunately, dispatches calls of
'gethostbyname' to a thread pool.

Without seeing the commit history, I'd assume the use of 'socket' and
threads by default is an artifact that predates the implementation of the
DNS protocol in Twisted. Twisted has, in 'twisted.names' [3], a DNS
protocol that uses UDP and leverages the reactor appropriately. Thankfully,
Twisted has a reactor method called 'installResolver' [4] that allows you
to hook in any DNS resolver implementation you want so you aren't stuck
using the default, threaded implementation.

As far as asyncio, it also defaults to an implementation that delegates to
an executor (default: threadpool). Unlike Twisted, though, it appears to
require a subclass of the event loop to override the 'getaddrinfo' method
[5].

[1]
https://github.com/twisted/twisted/blob/trunk/twisted/internet/tcp.py#L622
[2]
https://github.com/twisted/twisted/blob/trunk/twisted/internet/base.py#L257
[3] https://github.com/twisted/twisted/tree/trunk/twisted/names
[4]
https://github.com/twisted/twisted/blob/trunk/twisted/internet/base.py#L509
[5]
https://github.com/python/cpython/blob/master/Lib/asyncio/base_events.py#L572

On Sat, Feb 20, 2016, 03:31 Marko Rauhamaa <marko@pacujo.net> wrote:

> Paul Rubin <no.email@nospam.invalid>:
>
> > I've just felt depressed whenever I've looked at any Python async
> > stuff. I've written many Python programs with threads and not gotten
> > into the trouble that people keep warning about.
>
> Programming-model-wise, asyncio is virtually identical with threads. In
> each, I dislike the implicit state concept. I want the state to stand
> out with big block letters.
>
> > I've just felt depressed whenever I've looked at any Python async
> > stuff. I've written many Python programs with threads and not gotten
> > into the trouble that people keep warning about. But I haven't really
> > understood the warnings, so maybe they know something I don't. I just
> > write in a multiprocessing style, with every mutable object owned by
> > exactly one thread and accessed only by RPC through queues, sort of a
> > poor man's Erlang. There's a performance hit but there's a much bigger
> > one from using Python in the first place, so I just live with it.
>
> Good for you if you have been able to choose your own programming model.
> Most people have to deal with a legacy mess. Also, maintainers who
> inherit your tidy code might not be careful to ship only nonmutable
> objects in the queues.
>
> Your way of using threads works, of course, with the caveat that it is
> not possible to get rid of a blocking thread from the outside. With
> asyncio, you can at least cancel tasks.
>
>
> Marko
> --
> https://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

#103271

Hello there,

I realize that this discussion of supporting asynchronous name 
lookup requests in DNS is merely a detour in this thread on asyncio, 
but I couldn't resist mentioning an existing tool.

>> getaddrinfo is a notorious pain but I think it's just a library 
>> issue; an async version should be possible in principle.  How 
>> does Twisted handle it?  Does it have a version?
>
>In a (non-Python) program of mine, I got annoyed by synchronous 
>name lookups, so I hacked around it: instead of using the regular 
>library functions, I just do a DNS lookup directly (which can then 
>be event-based - send a UDP packet, get notified when a UDP packet 
>arrives). Downside: Ignores /etc/nsswitch.conf and /etc/hosts, and 
>goes straight to the name server. Upside: Is able to do its own 
>caching, since the DNS library gives me the TTLs, but 
>gethostbyname/getaddrinfo won't.

Another (non-Python) DNS name lookup library that does practically 
the same thing (along with the shortcomingsn you mentioned, Chris: 
no NSS nor /etc/hosts) is the adns library.  Well, it is DNS, after 
all.

  http://www.gnu.org/software/adns/
  https://pypi.python.org/pypi/adns-python/1.2.1

And, there are Python bindings.  I have been quite happy using the 
adns tools (and tools built on the Python bindings) for mass lookups 
(millions of DNS names).  It works very nicely.

Just sharing knowledge of an existing tool,

-Martin

-- 
Martin A. Brown
http://linux-ip.net/

[toc] | [prev] | [next] | [standalone]

#103278

On Sun, Feb 21, 2016 at 4:45 AM, Martin A. Brown <martin@linux-ip.net> wrote:
> Another (non-Python) DNS name lookup library that does practically
> the same thing (along with the shortcomingsn you mentioned, Chris:
> no NSS nor /etc/hosts) is the adns library.  Well, it is DNS, after
> all.
>
>   http://www.gnu.org/software/adns/
>   https://pypi.python.org/pypi/adns-python/1.2.1
>
> And, there are Python bindings.  I have been quite happy using the
> adns tools (and tools built on the Python bindings) for mass lookups
> (millions of DNS names).  It works very nicely.
>
> Just sharing knowledge of an existing tool,
>

Ultimately, anything that replaces a gethostbyname/getaddrinfo call
with an explicit DNS lookup is going to have the exact same benefit
and downside: lookups won't freeze the program, but you can't use
/etc/hosts any more. (Slightly sloppy language but that's how an end
user will see it.)

ChrisA

[toc] | [prev] | [next] | [standalone]

#103015

On Wed, Feb 17, 2016 at 2:21 AM, Frank Millman <frank@chagford.com> wrote:
> I would love to drive the database asynchronously, but of the three
> databases I use, only psycopg2 seems to have asyncio support. As my
> home-grown solution (using queues) seems to be working well so far, I am
> sticking with that until I start to experience responsiveness issues. If
> that happens, my first line of attack will be to switch from threads to
> processes.

And this is where we demonstrate divergent thought processes. *My*
first line of attack if hybrid async/thread doesn't work would be to
mandate a PostgreSQL backend, not to switch to hybrid async/process :)
Is the added value of "you get three options of database back-end"
worth the added cost of "but now my code is massively more complex"?

ChrisA

[toc] | [prev] | [next] | [standalone]

#103016

"Chris Angelico"  wrote in message 
news:CAPTjJmqMiE4gROqNYVHwAhCn2mWQeYQxt5KvFivotrHQY-sNdw@mail.gmail.com...
>
> On Wed, Feb 17, 2016 at 2:21 AM, Frank Millman <frank@chagford.com> wrote:
> > I would love to drive the database asynchronously, but of the three
> > databases I use, only psycopg2 seems to have asyncio support. As my
> > home-grown solution (using queues) seems to be working well so far, I am
> > sticking with that until I start to experience responsiveness issues. If
> > that happens, my first line of attack will be to switch from threads to
> > processes.
>
> And this is where we demonstrate divergent thought processes. *My*
> first line of attack if hybrid async/thread doesn't work would be to
> mandate a PostgreSQL backend, not to switch to hybrid async/process :)
> Is the added value of "you get three options of database back-end"
> worth the added cost of "but now my code is massively more complex"?

Then we will have to agree to diverge ;-)

If I ever get my app off the ground, it will be an all-purpose, 
multi-company, multi-currency, multi-everything accounting/business system.

There is a massive market out there, and a large percentage of that is 
Microsoft-only shops. I have no intention of cutting myself off from that 
market before I even start.

I am very happy with my choice of 3 databases -

1. sqlite3 - ideal for demo purposes and for one-man businesses

2. Sql Server for those that insist on it

3. PostgreSQL for every one else, and my recommendation if asked

Frank

[toc] | [prev] | [next] | [standalone]

#103008

"Kevin Conway"  wrote in message 
news:CAKF=+dim8wzPRvm86_V2W5-XSopCcHvgm0Hy8r4xeHDYzy_P4w@mail.gmail.com...
>
> If you're handling coroutines there is an asyncio facility for "background
> tasks". The ensure_future [1] will take a coroutine, attach it to a Task,
> and return a future to you that resolves when the coroutine is complete.
> The coroutine you schedule with that function will not cause your current
> coroutine to wait unless you await the future it returns.
>
> [1]
> https://docs.python.org/3/library/asyncio-task.html#asyncio.ensure_future
>

Thank you Kevin!

That works perfectly, and is much neater than my effort.

Frank

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.python

csiph-web