Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: "Frank Millman" <frank@chagford.com>
Newsgroups: comp.lang.python
Subject: Re: Question about asyncio and blocking operations
Date: Wed, 27 Jan 2016 16:40:29 +0200
Lines: 86
Message-ID: <mailman.41.1453956402.2338.python-list@python.org>
References: <n8038j$575$1@ger.gmane.org> <n8818q$35e$1@ger.gmane.org> <CALwzidk-RBkB-vi6CgcEeoFHQrsoTFvqX9MqzDD=rnY5bOCRUg@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original
Content-Transfer-Encoding: 7bit
In-Reply-To: <CALwzidk-RBkB-vi6CgcEeoFHQrsoTFvqX9MqzDD=rnY5bOCRUg@mail.gmail.com>
Importance: Normal
Precedence: list
Xref: csiph.com comp.lang.python:102169

"Ian Kelly"  wrote in message 
news:CALwzidk-RBkB-vi6CgcEeoFHQrsoTFvqX9MqzDD=rnY5bOCRUg@mail.gmail.com...

> On Tue, Jan 26, 2016 at 7:15 AM, Frank Millman <frank@chagford.com> wrote:
> >
> > If I return the cursor, I can iterate over it, but isn't this a blocking
> > operation? As far as I know, the DB adaptor will only actually retrieve 
> > the
> > row when requested.
> >
> > If I am right, I should call fetchall() while inside get_rows(), and 
> > return
> > all the rows as a list.
> >
>
> You probably want an asynchronous iterator here. If the cursor doesn't
> provide that, then you can wrap it in one. In fact, this is basically
> one of the examples in the PEP:
> https://www.python.org/dev/peps/pep-0492/#example-1
>

Thanks, Ian. I had a look, and it does seem to fit the bill, but I could not 
get it to work, and I am running out of time.

Specifically, I tried to get it working with the sqlite3 cursor. I am no 
expert, but after some googling I tried this -

import sqlite3
conn = sqlite3.connect('/sqlite_db')
cur = conn.cursor()

async def __aiter__(self):
    return self

async def __anext__(self):
    loop = asyncio.get_event_loop()
    return await loop.run_in_executor(None, self.__next__)

import types
cur.__aiter__ = types.MethodType( __aiter__, cur )
cur.__anext__ = types.MethodType( __anext__, cur )

It failed with this exception -

AttributeError: 'sqlite3.Cursor' object has no attribute '__aiter__'

I think this is what happens if a class uses 'slots' to define its 
attributes - it will not permit the creation of a new one.

Anyway, moving on, I decided to change tack. Up to now I have been trying to 
isolate the function where I actually communicate with the database, and 
wrap that in a Future with 'run_in_executor'.

In practice, the vast majority of my interactions with the database consist 
of very small CRUD commands, and will have minimal impact on response times 
even if they block. So I decided to focus on a couple of functions which are 
larger, and try to wrap the entire function in a Future with 
'run_in_executor'.

It seems to be working, but it looks a bit odd, so I will show what I am 
doing and ask for feedback.

Assume a slow function -

async def slow_function(arg1, arg2):
    [do stuff]

It now looks like this -

async def slow_function(arg1, arg2):
    loop = asyncio.get_event_loop()
    await loop.run_in_executor(None, slow_function_1, arg1, arg2)

def slow_function_1(self, arg1, arg2):
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    loop.run_until_complete(slow_function_2(arg1, arg2))

async slow_function_2(arg1, arg2):
    [do stuff]

Does this look right?

Frank