Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #102786 > unrolled thread

Re: asyncio and blocking - an update

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: asyncio and blocking - an update "Frank Millman" <frank@chagford.com> - 2016-02-11 07:45 +0200

#102786 — Re: asyncio and blocking - an update

"Frank Millman"  wrote in message news:n9c4p3$gmp$1@ger.gmane.org...
>
> Some of you may have been following my attempts to modify my asyncio app 
> so that it does not block when accessing the database. Here is an update.
>

Here is an update to my update ...

> I came up with what felt like a good idea. Run the database handler in a 
> separate thread, pass requests to it using a queue.Queue, and get it to 
> pass results back using an asyncio.Queue.
>
> It works, but I had a vague sense that performance was a bit sluggish, so 
> I tried the 'recommended' approach of using asyncio.run_in_executor() to 
> execute database calls in a separate thread. It felt a bit faster.
>
> Now I have written a proper timing test, and the recommended approach is 
> much faster. I am not 100% sure of the reason, but I think the problem is 
> that, with my method, when the database tries to 'put' a row on the return 
> queue, it has to use 'loop.call_soon_threadsafe()', and this seems to 
> create a bottleneck.
>

I have come up with a plan that seems to provide a solution.

Instead of 'putting' one row at a time, let the database handler build up a 
block of rows, and then 'put' the block.

I tried a block of 10, and it ran a lot faster. I increased it to 50, and it 
ran faster again. I tried 100 and there was not much improvement, so 50 
seems like an optimum number. The speed is now only slightly slower than 
run_in_executor(), and it is more truly asynchronous.

Frank

[toc] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web