Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #102192 > unrolled thread

Re: Question about asyncio and blocking operations

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Question about asyncio and blocking operations "Frank Millman" <frank@chagford.com> - 2016-01-28 18:40 +0200

#102192 — Re: Question about asyncio and blocking operations

"Ian Kelly"  wrote in message 
news:CALwzidnGbz7kM=D7MKua2tA9-csFn9u0OHL0w-X5Bbixpcw4Ow@mail.gmail.com...

> On Jan 28, 2016 4:13 AM, "Frank Millman" <frank@chagford.com> wrote:
> >
> > I *think* I have this one covered. When the caller makes a request, it
> creates an instance of an asyncio.Queue, and includes it with the request.
> The db handler uses this queue to send the result back.
> >
> > Do you see any problem with this?
>
> That seems reasonable to me. I assume that when you send the result back
> you would be queuing up individual rows and not just sending a single
> object across, which could be more easily with just a single future.
>

I have hit a snag. It feels like a bug in 'await q.get()', though I am sure
it is just me misunderstanding how it works.

I can post some working code if necessary, but here is a short description.

Here is the database handler - 'request_queue' is a queue.Queue -

        while not request_queue.empty():
            return_queue, sql = request_queue.get()
            cur.execute(sql)
            for row in cur:
                return_queue.put_nowait(row)
             return_queue.put_nowait(None)
            request_queue.task_done()

The caller requests some data from the database like this.

    return_queue = asyncio.Queue()
    sql = 'SELECT ...'
    request_queue.put((return_queue, sql))
    while True:
        row = await return_queue.get()
        if row is None:
            break
        print('got', row)
    return_queue.task_done()

The first time 'await return_queue.get()' is called, return_queue is empty,
as the db handler has not had time to do anything yet.

It is supposed to pause there, wait for something to appear in the queue,
and then process it. I have confirmed that the db handler populates the
queue virtually instantly.

What seems to happen is that it pauses there, but then waits for some other
event in the event loop to occur before it continues. Then it processes all
rows very quickly.

I am running a 'counter' task in the background that prints a sequential
number, using await asyncio.sleep(1). I noticed a short but variable delay
before the rows were printed, and thought it might be waiting for the
counter. I increased the sleep to 10, and sure enough it now waits up to 10
seconds before printing any rows.

Any ideas?

Frank

[toc] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web