Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #102941 > unrolled thread

asyncio - run coroutine in the background

Back to article view | Back to comp.lang.python

Contents

  asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-15 08:35 +0200
    Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-15 08:54 +0200
      Re: asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-15 09:16 +0200
        Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-15 09:34 +0200
        Re: asyncio - run coroutine in the background Paul Rubin <no.email@nospam.invalid> - 2016-02-14 23:39 -0800
          Re: asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-15 10:17 +0200
          Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-15 13:05 +0200
          Re: asyncio - run coroutine in the background Chris Angelico <rosuav@gmail.com> - 2016-02-16 16:51 +1100
          Re: asyncio - run coroutine in the background Kevin Conway <kevinjacobconway@gmail.com> - 2016-02-16 13:22 +0000
            Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 16:17 +0200
              Re: asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-16 16:36 +0200
              Re: asyncio - run coroutine in the background Kevin Conway <kevinjacobconway@gmail.com> - 2016-02-16 14:54 +0000
              Re: asyncio - run coroutine in the background Steven D'Aprano <steve@pearwood.info> - 2016-02-17 02:17 +1100
                Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 19:12 +0200
                  Re: asyncio - run coroutine in the background Paul Rubin <no.email@nospam.invalid> - 2016-02-17 20:38 -0800
                    Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-18 08:10 +0200
                Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 19:13 +0200
                Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 19:14 +0200
                Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 19:15 +0200
                  Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 19:15 +0200
                    Re: asyncio - run coroutine in the background Robin Becker <robin@reportlab.com> - 2016-02-16 17:52 +0000
              Re: asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-16 17:21 +0200
                Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-16 19:20 +0200
                  Re: asyncio - run coroutine in the background Paul Rubin <no.email@nospam.invalid> - 2016-02-19 23:40 -0800
                    Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-20 10:13 +0200
                      Re: asyncio - run coroutine in the background Paul Rubin <no.email@nospam.invalid> - 2016-02-20 00:37 -0800
                        Re: asyncio - run coroutine in the background Chris Angelico <rosuav@gmail.com> - 2016-02-20 19:52 +1100
                          Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-20 10:59 +0200
                            Re: asyncio - run coroutine in the background Chris Angelico <rosuav@gmail.com> - 2016-02-20 20:02 +1100
                        Re: asyncio - run coroutine in the background Marko Rauhamaa <marko@pacujo.net> - 2016-02-20 11:28 +0200
                          Re: asyncio - run coroutine in the background Kevin Conway <kevinjacobconway@gmail.com> - 2016-02-20 13:52 +0000
                        Re: asyncio - run coroutine in the background "Martin A. Brown" <martin@linux-ip.net> - 2016-02-20 09:45 -0800
                        Re: asyncio - run coroutine in the background Chris Angelico <rosuav@gmail.com> - 2016-02-21 08:47 +1100
              Re: asyncio - run coroutine in the background Chris Angelico <rosuav@gmail.com> - 2016-02-17 02:28 +1100
              Re: asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-16 17:45 +0200
          Re: asyncio - run coroutine in the background "Frank Millman" <frank@chagford.com> - 2016-02-16 15:52 +0200

Page 1 of 2  [1] 2  Next page →

#102941 — asyncio - run coroutine in the background

Hi all

Using asyncio, there are times when I want to execute a coroutine which is 
time-consuming. I do not need the result immediately, and I do not want to 
block the current task, so I want to run it in the background.

run_in_executor() can run an arbitrary function in the background, but a 
coroutine needs an event loop. After some experimenting I came up with 
this -

class BackgroundTask:
    async def run(self, coro, args, callback=None):
        loop = asyncio.get_event_loop()
        loop.run_in_executor(None, self.task_runner, coro, args, callback)

    def task_runner(self, coro, args, callback):
        loop = asyncio.new_event_loop()
        asyncio.set_event_loop(loop)

        fut = asyncio.ensure_future(coro(*args))
        if callback is not None:
            fut.add_done_callback(callback)

        loop.run_until_complete(fut)
        loop.close()

Usage -

    bg_task = BackgroundTask()
    args = (arg1, arg2 ...)
    callback = my_callback_function
    await bg_task.run(coro, args, callback)

Although it 'awaits' bk_task.run(), it returns immediately, as it is simply 
waiting for run_in_executor() to be launched.

Hope this is of some interest.

Frank Millman

[toc] | [next] | [standalone]

#102942

"Frank Millman" <frank@chagford.com>:

> Using asyncio, there are times when I want to execute a coroutine which
> is time-consuming. I do not need the result immediately, and I do not
> want to block the current task, so I want to run it in the background.

You can't run code "in the background" using asyncio. Coroutines perform
cooperative multitasking in a single thread on a single CPU.

Parallel processing requires the use of threads or, often preferably,
processes.

To put it in another way, never run time-consuming code in asyncio.


Marko

[toc] | [prev] | [next] | [standalone]

#102943

"Marko Rauhamaa"  wrote in message news:8737sumpjl.fsf@elektro.pacujo.net...
>
> "Frank Millman" <frank@chagford.com>:
>
> > Using asyncio, there are times when I want to execute a coroutine which
> > is time-consuming. I do not need the result immediately, and I do not
> > want to block the current task, so I want to run it in the background.
>
> You can't run code "in the background" using asyncio. Coroutines perform
> cooperative multitasking in a single thread on a single CPU.
>
> Parallel processing requires the use of threads or, often preferably,
> processes.
>
> To put it in another way, never run time-consuming code in asyncio.
>

No arguments there.

I started with a task that ran quickly, but as I added stuff it started to 
slow down.

The execution of the task involves calling some existing functions, which 
are themselves coroutines. As you have noted elsewhere, once you turn one 
function into a coroutine, all calls higher up the chain have to be 
coroutines as well.

The benefit of my class is that it enables me to take the coroutine and run 
it in another thread, without having to re-engineer the whole thing.

Hope this makes sense.

Frank

[toc] | [prev] | [next] | [standalone]

#102944

"Frank Millman" <frank@chagford.com>:

> The benefit of my class is that it enables me to take the coroutine
> and run it in another thread, without having to re-engineer the whole
> thing.
>
> Hope this makes sense.

Sure.


Marko

[toc] | [prev] | [next] | [standalone]

#102946

"Frank Millman" <frank@chagford.com> writes:
> The benefit of my class is that it enables me to take the coroutine
> and run it in another thread, without having to re-engineer the whole
> thing.

Threads in Python don't get you parallelism either, of course.

I haven't used async/await yet and it's looking painful.  I've been
wanting to read this:

 http://www.snarky.ca/how-the-heck-does-async-await-work-in-python-3-5

but I start to think it isn't all that great an approach to concurrency.

[toc] | [prev] | [next] | [standalone]

#102951

"Paul Rubin"  wrote in message 
news:87h9ha8lt0.fsf@jester.gateway.pace.com...
>
> "Frank Millman" <frank@chagford.com> writes:
> > The benefit of my class is that it enables me to take the coroutine
> > and run it in another thread, without having to re-engineer the whole
> > thing.
>
> Threads in Python don't get you parallelism either, of course.
>

Agreed. My class does not alter the total time taken, but it does free up 
the original task to carry on with other work.

run_in_executor() uses threads by default, but it does allow you to specify 
processes as an alternative.

> I haven't used async/await yet and it's looking painful.  I've been
> wanting to read this:
>
>  http://www.snarky.ca/how-the-heck-does-async-await-work-in-python-3-5
>
> but I start to think it isn't all that great an approach to concurrency.
>

Thanks for that link. I had a quick scan, and it looks interesting, but some 
of it a bit above my head. I have bookmarked it, as I think that as my 
understanding increases, I will gain more from it on each re-read.

Frank

[toc] | [prev] | [next] | [standalone]

#102959

Paul Rubin <no.email@nospam.invalid>:

> Threads in Python don't get you parallelism either, of course.

Ah, of course.

Processes it is, then.


Marko

[toc] | [prev] | [next] | [standalone]

#102988

On Mon, Feb 15, 2016 at 6:39 PM, Paul Rubin <no.email@nospam.invalid> wrote:
> "Frank Millman" <frank@chagford.com> writes:
>> The benefit of my class is that it enables me to take the coroutine
>> and run it in another thread, without having to re-engineer the whole
>> thing.
>
> Threads in Python don't get you parallelism either, of course.
>

They can. The only limitation is that, in CPython (and some others),
no two threads can concurrently be executing Python byte-code. The
instant you drop into a C-implemented function, it can release the GIL
and let another thread start running. Obviously this happens any time
there's going to be a blocking API call (eg if one thread waits on a
socket read, others can run), but it can also happen with
computational work:

import numpy
import threading

def thread1():
    arr = numpy.zeros(100000000, dtype=numpy.int64)
    while True:
        print("1: %d" % arr[0])
        arr += 1
        arr = (arr * arr) % 142957

def thread2():
    arr = numpy.zeros(100000000, dtype=numpy.int64)
    while True:
        print("2: %d" % arr[0])
        arr += 2
        arr = (arr * arr) % 142957

threading.Thread(target=thread1).start()
thread2()

This will happily keep two CPU cores occupied. Most of the work is
being done inside Numpy, which releases the GIL before doing any work.
So it's not strictly true that threading can't parallelise Python code
(and as mentioned, it depends on your interpreter - Jython can, I
believe, do true multithreading), but just that there are limitations
on what can execute concurrently.

ChrisA

[toc] | [prev] | [next] | [standalone]

#103005

If you're handling coroutines there is an asyncio facility for "background
tasks". The ensure_future [1] will take a coroutine, attach it to a Task,
and return a future to you that resolves when the coroutine is complete.
The coroutine you schedule with that function will not cause your current
coroutine to wait unless you await the future it returns.

[1]
https://docs.python.org/3/library/asyncio-task.html#asyncio.ensure_future

On Mon, Feb 15, 2016, 23:53 Chris Angelico <rosuav@gmail.com> wrote:

> On Mon, Feb 15, 2016 at 6:39 PM, Paul Rubin <no.email@nospam.invalid>
> wrote:
> > "Frank Millman" <frank@chagford.com> writes:
> >> The benefit of my class is that it enables me to take the coroutine
> >> and run it in another thread, without having to re-engineer the whole
> >> thing.
> >
> > Threads in Python don't get you parallelism either, of course.
> >
>
> They can. The only limitation is that, in CPython (and some others),
> no two threads can concurrently be executing Python byte-code. The
> instant you drop into a C-implemented function, it can release the GIL
> and let another thread start running. Obviously this happens any time
> there's going to be a blocking API call (eg if one thread waits on a
> socket read, others can run), but it can also happen with
> computational work:
>
> import numpy
> import threading
>
> def thread1():
>     arr = numpy.zeros(100000000, dtype=numpy.int64)
>     while True:
>         print("1: %d" % arr[0])
>         arr += 1
>         arr = (arr * arr) % 142957
>
> def thread2():
>     arr = numpy.zeros(100000000, dtype=numpy.int64)
>     while True:
>         print("2: %d" % arr[0])
>         arr += 2
>         arr = (arr * arr) % 142957
>
> threading.Thread(target=thread1).start()
> thread2()
>
> This will happily keep two CPU cores occupied. Most of the work is
> being done inside Numpy, which releases the GIL before doing any work.
> So it's not strictly true that threading can't parallelise Python code
> (and as mentioned, it depends on your interpreter - Jython can, I
> believe, do true multithreading), but just that there are limitations
> on what can execute concurrently.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

#103009

Kevin Conway <kevinjacobconway@gmail.com>:

> If you're handling coroutines there is an asyncio facility for
> "background tasks". The ensure_future [1] will take a coroutine,
> attach it to a Task, and return a future to you that resolves when the
> coroutine is complete.

Ok, yes, but those "background tasks" monopolize the CPU once they are
scheduled to run.

If your "background task" doesn't need a long time to run, just call the
function in the foreground and be done with it. If it does consume time,
you need to delegate it to a separate process so the other tasks remain
responsive.


Marko

[toc] | [prev] | [next] | [standalone]

#103010

"Marko Rauhamaa"  wrote in message news:87d1rwpwo2.fsf@elektro.pacujo.net...
>
> Kevin Conway <kevinjacobconway@gmail.com>:
>
> > If you're handling coroutines there is an asyncio facility for
> > "background tasks". The ensure_future [1] will take a coroutine,
> > attach it to a Task, and return a future to you that resolves when the
> > coroutine is complete.
>
> Ok, yes, but those "background tasks" monopolize the CPU once they are
> scheduled to run.
>
> If your "background task" doesn't need a long time to run, just call the
> function in the foreground and be done with it. If it does consume time,
> you need to delegate it to a separate process so the other tasks remain
> responsive.
>

I will explain my situation - perhaps you can tell me if it makes sense.

My background task does take a long time to run - about 10 seconds - but 
most of that time is spent waiting for database responses, which is handled 
in another thread.

You could argue that the database thread should rather be handled by another 
process, and that is definitely an option if I find that response times are 
affected.

So far my response times have been very good, even with database activity in 
the background. However, I have not simulated a large number of concurrent 
users. That could throw up the kinds of problem that you are concerned 
about.

Frank

[toc] | [prev] | [next] | [standalone]

#103011

> Ok, yes, but those "background tasks" monopolize the CPU once they are
scheduled to run.

This is true if the coroutines are cpu bound. If that is the case then a
coroutine is likely the wrong choice for that code to begin with.
Coroutines, in asyncio land, are primarily designed for io bound work.

> My background task does take a long time to run - about 10 seconds - but
most of that time is spent waiting for database responses, which is handled
in another thread.

Something else to look into is an asyncio driver for your database
connections. Threads aren't inherently harmful, but using them to achieve
async networking when running asyncio is a definite code smell since that
is precisely the problem asyncio is supposed to solve for.

On Tue, Feb 16, 2016, 08:37 Frank Millman <frank@chagford.com> wrote:

> "Marko Rauhamaa"  wrote in message news:87d1rwpwo2.fsf@elektro.pacujo.net.
> ..
> >
> > Kevin Conway <kevinjacobconway@gmail.com>:
> >
> > > If you're handling coroutines there is an asyncio facility for
> > > "background tasks". The ensure_future [1] will take a coroutine,
> > > attach it to a Task, and return a future to you that resolves when the
> > > coroutine is complete.
> >
> > Ok, yes, but those "background tasks" monopolize the CPU once they are
> > scheduled to run.
> >
> > If your "background task" doesn't need a long time to run, just call the
> > function in the foreground and be done with it. If it does consume time,
> > you need to delegate it to a separate process so the other tasks remain
> > responsive.
> >
>
> I will explain my situation - perhaps you can tell me if it makes sense.
>
> My background task does take a long time to run - about 10 seconds - but
> most of that time is spent waiting for database responses, which is handled
> in another thread.
>
> You could argue that the database thread should rather be handled by
> another
> process, and that is definitely an option if I find that response times are
> affected.
>
> So far my response times have been very good, even with database activity
> in
> the background. However, I have not simulated a large number of concurrent
> users. That could throw up the kinds of problem that you are concerned
> about.
>
> Frank
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

#103013

On Wed, 17 Feb 2016 01:17 am, Marko Rauhamaa wrote:

> Ok, yes, but those "background tasks" monopolize the CPU once they are
> scheduled to run.

Can you show some code demonstrating this?



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#103018

Steven D'Aprano <steve@pearwood.info>:

> On Wed, 17 Feb 2016 01:17 am, Marko Rauhamaa wrote:
>
>> Ok, yes, but those "background tasks" monopolize the CPU once they
>> are scheduled to run.
>
> Can you show some code demonstrating this?

Sure:

========================================================================
#!/usr/bin/env python3

import asyncio, time

def main():
    asyncio.get_event_loop().run_until_complete(asyncio.wait([
        background_task(),
        looping_task() ]))

@asyncio.coroutine
def looping_task():
    while True:
        yield from asyncio.sleep(1)
        print(int(time.time()))

@asyncio.coroutine
def background_task():
    yield from asyncio.sleep(4)
    t = time.time()
    while time.time() - t < 10:
        pass

if __name__ == '__main__':
    main()
========================================================================

which prints:

   1455642629
   1455642630
   1455642631
   1455642642                <============== gap
   1455642643
   1455642644
   1455642645


Marko

[toc] | [prev] | [next] | [standalone]

#103082

Marko Rauhamaa <marko@pacujo.net> writes:
> @asyncio.coroutine
> def background_task(): ...
>     while time.time() - t < 10:
>         pass

Wait, that's a cpu-busy loop, you can't do that in cooperative
multitasking.  Of course you need a wait there.

[toc] | [prev] | [next] | [standalone]

#103086

Paul Rubin <no.email@nospam.invalid>:

> Marko Rauhamaa <marko@pacujo.net> writes:
>> @asyncio.coroutine
>> def background_task(): ...
>>     while time.time() - t < 10:
>>         pass
>
> Wait, that's a cpu-busy loop, you can't do that in cooperative
> multitasking.  Of course you need a wait there.

That was the very point: to demonstrate that coroutines monopolize the
CPU.


Marko

[toc] | [prev] | [next] | [standalone]

#103019

Steven D'Aprano <steve@pearwood.info>:

> On Wed, 17 Feb 2016 01:17 am, Marko Rauhamaa wrote:
>
>> Ok, yes, but those "background tasks" monopolize the CPU once they are
>> scheduled to run.
>
> Can you show some code demonstrating this?

Sure:

========================================================================
#!/usr/bin/env python3

import asyncio, time

def main():
    asyncio.get_event_loop().run_until_complete(asyncio.wait([
        background_task(),
        looping_task() ]))

@asyncio.coroutine
def looping_task():
    while True:
        yield from asyncio.sleep(1)
        print(int(time.time()))

@asyncio.coroutine
def background_task():
    yield from asyncio.sleep(4)
    t = time.time()
    while time.time() - t < 10:
        pass

if __name__ == '__main__':
    main()
========================================================================

which prints:

   1455642629
   1455642630
   1455642631
   1455642642                <============== gap
   1455642643
   1455642644
   1455642645


Marko

[toc] | [prev] | [next] | [standalone]

#103020

Steven D'Aprano <steve@pearwood.info>:

> On Wed, 17 Feb 2016 01:17 am, Marko Rauhamaa wrote:
>
>> Ok, yes, but those "background tasks" monopolize the CPU once they are
>> scheduled to run.
>
> Can you show some code demonstrating this?

Sure:

========================================================================
#!/usr/bin/env python3

import asyncio, time

def main():
    asyncio.get_event_loop().run_until_complete(asyncio.wait([
        background_task(),
        looping_task() ]))

@asyncio.coroutine
def looping_task():
    while True:
        yield from asyncio.sleep(1)
        print(int(time.time()))

@asyncio.coroutine
def background_task():
    yield from asyncio.sleep(4)
    t = time.time()
    while time.time() - t < 10:
        pass

if __name__ == '__main__':
    main()
========================================================================

which prints:

   1455642629
   1455642630
   1455642631
   1455642642                <============== gap
   1455642643
   1455642644
   1455642645


Marko

[toc] | [prev] | [next] | [standalone]

#103021

Steven D'Aprano <steve@pearwood.info>:

> On Wed, 17 Feb 2016 01:17 am, Marko Rauhamaa wrote:
>
>> Ok, yes, but those "background tasks" monopolize the CPU once they are
>> scheduled to run.
>
> Can you show some code demonstrating this?

Sure:

========================================================================
#!/usr/bin/env python3

import asyncio, time

def main():
    asyncio.get_event_loop().run_until_complete(asyncio.wait([
        background_task(),
        looping_task() ]))

@asyncio.coroutine
def looping_task():
    while True:
        yield from asyncio.sleep(1)
        print(int(time.time()))

@asyncio.coroutine
def background_task():
    yield from asyncio.sleep(4)
    t = time.time()
    while time.time() - t < 10:
        pass

if __name__ == '__main__':
    main()
========================================================================

which prints:

   1455642629
   1455642630
   1455642631
   1455642642
   1455642643
   1455642644
   1455642645


Marko

[toc] | [prev] | [next] | [standalone]

#103022

Marko Rauhamaa <marko@pacujo.net>:

> Sure:

Sorry for the multiple copies.


Marko

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python

csiph-web