Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: "Frank Millman" <frank@chagford.com>
Newsgroups: comp.lang.python
Subject: Re: Question about asyncio and blocking operations
Date: Thu, 28 Jan 2016 18:40:56 +0200
Lines: 65
Message-ID: <mailman.62.1453999322.2338.python-list@python.org>
References: <n8038j$575$1@ger.gmane.org> <n8818q$35e$1@ger.gmane.org> <CALwzidk-RBkB-vi6CgcEeoFHQrsoTFvqX9MqzDD=rnY5bOCRUg@mail.gmail.com> <n8aln3$fah$1@ger.gmane.org> <CALwzidn6TvN9W-2qnn2JYvJu8NHzn499nPtfjn9OHjdDcebVbA@mail.gmail.com> <CALwzidmmUeL3Sq9o6ZgKYM3-V22HCshG03wQqsXvhh6xy1otgQ@mail.gmail.com> <CALwzidkr-fT6S6wH2caNaxyQvUdAw=x7XDqkqOfNRrwzwnjvOQ@mail.gmail.com> <n8cm4k$ojk$1@ger.gmane.org> <CAPTjJmr162+K4LZeFpXruR6wxrHxbR-_wkrCLLDyR7kST+kjYg@mail.gmail.com> <n8ct18$fu4$1@ger.gmane.org> <CALwzidnGbz7kM=D7MKua2tA9-csFn9u0OHL0w-X5Bbixpcw4Ow@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original
Content-Transfer-Encoding: 7bit
In-Reply-To: <CALwzidnGbz7kM=D7MKua2tA9-csFn9u0OHL0w-X5Bbixpcw4Ow@mail.gmail.com>
Importance: Normal
Precedence: list
Xref: csiph.com comp.lang.python:102192

"Ian Kelly"  wrote in message 
news:CALwzidnGbz7kM=D7MKua2tA9-csFn9u0OHL0w-X5Bbixpcw4Ow@mail.gmail.com...

> On Jan 28, 2016 4:13 AM, "Frank Millman" <frank@chagford.com> wrote:
> >
> > I *think* I have this one covered. When the caller makes a request, it
> creates an instance of an asyncio.Queue, and includes it with the request.
> The db handler uses this queue to send the result back.
> >
> > Do you see any problem with this?
>
> That seems reasonable to me. I assume that when you send the result back
> you would be queuing up individual rows and not just sending a single
> object across, which could be more easily with just a single future.
>

I have hit a snag. It feels like a bug in 'await q.get()', though I am sure
it is just me misunderstanding how it works.

I can post some working code if necessary, but here is a short description.

Here is the database handler - 'request_queue' is a queue.Queue -

        while not request_queue.empty():
            return_queue, sql = request_queue.get()
            cur.execute(sql)
            for row in cur:
                return_queue.put_nowait(row)
             return_queue.put_nowait(None)
            request_queue.task_done()

The caller requests some data from the database like this.

    return_queue = asyncio.Queue()
    sql = 'SELECT ...'
    request_queue.put((return_queue, sql))
    while True:
        row = await return_queue.get()
        if row is None:
            break
        print('got', row)
    return_queue.task_done()

The first time 'await return_queue.get()' is called, return_queue is empty,
as the db handler has not had time to do anything yet.

It is supposed to pause there, wait for something to appear in the queue,
and then process it. I have confirmed that the db handler populates the
queue virtually instantly.

What seems to happen is that it pauses there, but then waits for some other
event in the event loop to occur before it continues. Then it processes all
rows very quickly.

I am running a 'counter' task in the background that prints a sequential
number, using await asyncio.sleep(1). I noticed a short but variable delay
before the rows were printed, and thought it might be waiting for the
counter. I increased the sleep to 10, and sure enough it now waits up to 10
seconds before printing any rows.

Any ideas?

Frank