Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #88141 > unrolled thread

asyncio

Back to article view | Back to comp.lang.python

Contents

  asyncio Łukasz Ligowski <orangewarrior@gmail.com> - 2015-03-27 01:33 -0700
    Re: asyncio Marko Rauhamaa <marko@pacujo.net> - 2015-03-27 10:55 +0200
      Re: asyncio orangewarrior@gmail.com - 2015-03-27 02:32 -0700
        Re: asyncio Marko Rauhamaa <marko@pacujo.net> - 2015-03-27 14:38 +0200
          Re: asyncio Ian Kelly <ian.g.kelly@gmail.com> - 2015-03-27 08:15 -0600
            Re: asyncio Marko Rauhamaa <marko@pacujo.net> - 2015-03-27 17:44 +0200
              Re: asyncio Marko Rauhamaa <marko@pacujo.net> - 2015-03-27 17:52 +0200
            Re: asyncio orangewarrior@gmail.com - 2015-03-27 16:27 -0700

#88141 — asyncio

[Multipart message — attachments visible in raw view] — view raw

Hi,

I wrote simple asyncio program (see below) and I'm not sure if I understand
behavior correctly.
I have print_tasks coroutine which prints each task in a loop by using
Task.all_tasks function.
I have also task_launcher coroutine that launches (by loop.create_task())
simple task that waits some and prints "ping".
Question is why finished "ping" tasks keep accumulating in Task.all_tasks?
After a while there is a lot of:
<Task finished coro=<ping() done, defined at test.py:25> result=None>

when printing in Task.all_tasks and number of those increases as new ping
tasks are launched

Is there a way to prune finished tasks (I tried forcing gc) or I do
something wrong?

I ran on python 3.4.3.

Best regards,
L

import asyncio

loop = asyncio.get_event_loop()

@asyncio.coroutine
def print_tasks(loop):
        while True:
                tasks = asyncio.Task.all_tasks(loop)

                for task in tasks:
                        print(task)
                print()

                yield from asyncio.sleep(1)

@asyncio.coroutine
def task_launcher(loop):
        while True:
                yield from asyncio.sleep(1)
                loop.create_task(ping())

@asyncio.coroutine
def ping():
        yield from asyncio.sleep(2)
        print("ping")


loop.create_task(print_tasks(loop))
loop.create_task(task_launcher(loop))

loop.run_forever()

[toc] | [next] | [standalone]

#88143

Łukasz Ligowski <orangewarrior@gmail.com>:

> Is there a way to prune finished tasks (I tried forcing gc) or I do
> something wrong?

Guessing that you need to call asyncio.wait() or asyncio.wait_for() to
get rid of the zombies.

(Apparently the loop object keeps references to completed tasks. I'm not
sure this design choice is necessary.)


Marko

[toc] | [prev] | [next] | [standalone]

#88144

Hi,

On Friday, March 27, 2015 at 1:55:36 AM UTC-7, Marko Rauhamaa wrote:
> Łukasz Ligowski <orangewarrior@gmail.com>:
> 
> > Is there a way to prune finished tasks (I tried forcing gc) or I do
> > something wrong?
> 
> Guessing that you need to call asyncio.wait() or asyncio.wait_for() to
> get rid of the zombies.
> 
> (Apparently the loop object keeps references to completed tasks. I'm not
> sure this design choice is necessary.)

Should I launch tasks differently? Waiting for task that is done returns that task in done subset but it still lingers in list of all tasks...

Best regards,
L

[toc] | [prev] | [next] | [standalone]

#88145

orangewarrior@gmail.com:

> On Friday, March 27, 2015 at 1:55:36 AM UTC-7, Marko Rauhamaa wrote:
>> Łukasz Ligowski <orangewarrior@gmail.com>:
>> 
>> > Is there a way to prune finished tasks (I tried forcing gc) or I do
>> > something wrong?
>> 
>> Guessing that you need to call asyncio.wait() or asyncio.wait_for() to
>> get rid of the zombies.
>> 
>> (Apparently the loop object keeps references to completed tasks. I'm not
>> sure this design choice is necessary.)
>
> Should I launch tasks differently? Waiting for task that is done returns
> that task in done subset but it still lingers in list of all tasks...

I don't know the answer to your question. A superficial glance at the
relevant asyncio source code (and documentation) suggests you shouldn't
be seeing what you are seeing.

Tasks are kept in a weak set. Tasks should evaporate as soon as nobody
references them.

I wonder if there's a cyclical reference there somewhere. See if running
GC removes the tasks.


Marko

[toc] | [prev] | [next] | [standalone]

#88163

On Fri, Mar 27, 2015 at 6:38 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> I don't know the answer to your question. A superficial glance at the
> relevant asyncio source code (and documentation) suggests you shouldn't
> be seeing what you are seeing.
>
> Tasks are kept in a weak set. Tasks should evaporate as soon as nobody
> references them.

Actually I think this explains it. In the OP's while loop, he updates
his task list with the line:

tasks = asyncio.Task.all_tasks(loop)

This creates a strong reference to each of the returned tasks.  When
the loop comes back around, it calls all_tasks again, creating a
second strong reference to each task, then assigns the resulting set
to tasks, allowing the original set to be collected. At no point in
the loop is there ever not a strong reference to each task. So if
asyncio is relying on GC to prune its task set, they will never be
cleaned up.

To the OP: try deleting the tasks variable after you print it out. I
bet this will solve your problem.

[toc] | [prev] | [next] | [standalone]

#88166

Ian Kelly <ian.g.kelly@gmail.com>:

> Actually I think this explains it. In the OP's while loop, he updates
> his task list with the line:
>
> tasks = asyncio.Task.all_tasks(loop)
>
> This creates a strong reference to each of the returned tasks.

Good catch!


Marko

[toc] | [prev] | [next] | [standalone]

#88167

Marko Rauhamaa <marko@pacujo.net>:

> Ian Kelly <ian.g.kelly@gmail.com>:
>
>> Actually I think this explains it. In the OP's while loop, he updates
>> his task list with the line:
>>
>> tasks = asyncio.Task.all_tasks(loop)
>>
>> This creates a strong reference to each of the returned tasks.
>
> Good catch!

And demonstrates somewhat of a pitfall with weak references (which I
have never used).

Weak references are not as weak as they are thought to be.


Marko

[toc] | [prev] | [next] | [standalone]

#88188

On Friday, March 27, 2015 at 7:17:26 AM UTC-7, Ian wrote:
> On Fri, Mar 27, 2015 at 6:38 AM, Marko Rauhamaa  wrote:
> > I don't know the answer to your question. A superficial glance at the
> > relevant asyncio source code (and documentation) suggests you shouldn't
> > be seeing what you are seeing.
> >
> > Tasks are kept in a weak set. Tasks should evaporate as soon as nobody
> > references them.
> 
> Actually I think this explains it. In the OP's while loop, he updates
> his task list with the line:
> 
> tasks = asyncio.Task.all_tasks(loop)
> 
> This creates a strong reference to each of the returned tasks.  When
> the loop comes back around, it calls all_tasks again, creating a
> second strong reference to each task, then assigns the resulting set
> to tasks, allowing the original set to be collected. At no point in
> the loop is there ever not a strong reference to each task. So if
> asyncio is relying on GC to prune its task set, they will never be
> cleaned up.
> 
> To the OP: try deleting the tasks variable after you print it out. I
> bet this will solve your problem.

I missed that part. Thanks!

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web