Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail From: Chris Angelico Newsgroups: comp.lang.python Subject: Re: Question about asyncio and blocking operations Date: Thu, 28 Jan 2016 21:56:50 +1100 Lines: 43 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-Trace: news.uni-berlin.de NHHdTyKv6HZ869MdlBL56gaDMxPamYpsRyxSdEeSQCkA== Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.000 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'url:pypi': 0.03; 'handler': 0.04; 'subject:Question': 0.05; 'caller': 0.07; 'socket': 0.07; 'cc:addr:python-list': 0.09; 'block.': 0.09; 'handler.': 0.09; 'thread': 0.10; 'assume': 0.11; 'python.': 0.11; 'jan': 0.11; 'thu,': 0.15; '2016': 0.16; 'async': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'operation,': 0.16; 'pgsql': 0.16; 'put.': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; 'sequencing': 0.16; 'then?': 0.16; 'there?': 0.16; 'think?': 0.16; 'thread.': 0.16; 'wrote:': 0.16; 'instance,': 0.18; 'cc:2**0': 0.20; 'cc:addr:python.org': 0.20; 'fairly': 0.22; 'code.': 0.23; 'seems': 0.23; 'header:In-Reply- To:1': 0.24; 'requests': 0.25; 'sort': 0.25; 'module': 0.25; 'separate': 0.27; 'switch': 0.27; 'message-id:@mail.gmail.com': 0.27; 'converting': 0.27; 'idea': 0.28; 'looks': 0.29; '(maybe': 0.29; 'blocking': 0.29; 'description,': 0.29; 'queue': 0.29; 'received:209.85.213.174': 0.29; "i'm": 0.30; 'work.': 0.30; "i'd": 0.31; 'operations': 0.31; 'says': 0.32; '[1]': 0.32; 'maybe': 0.33; 'run': 0.33; 'url:python': 0.33; 'handle': 0.34; 'gets': 0.35; 'received:google.com': 0.35; 'could': 0.35; 'tasks': 0.35; 'quite': 0.35; 'something': 0.35; 'but': 0.36; 'should': 0.36; 'url:org': 0.36; 'received:209.85': 0.36; 'possible': 0.36; 'faster': 0.36; 'pm,': 0.36; 'subject:: ': 0.37; 'received:209.85.213': 0.37; 'received:209': 0.38; 'wrong': 0.38; 'test': 0.39; 'hello,': 0.40; 'waiting': 0.60; 'your': 0.60; 'back': 0.62; 'making': 0.62; 'results': 0.66; 'jobs': 0.67; 'risk': 0.68; 'feeling': 0.72; 'frank': 0.72; '"it': 0.84; 'chrisa': 0.84; 'gut': 0.84; 'mistaken': 0.84; 'to:none': 0.91 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-type; bh=AsB1xKHKpLw8KObYUYvj9O1np6ls5yLXy4gh3kLhAfw=; b=dYGa8l1IRINVUaiYWPqhtZHqsTzd6Qsb1GX2xosKaRps5TkljUz8OCo05+64S7/Jth 7FbAeNob6kzLFram4NuRAh5Z9HwXakZQmabLw2AUq72hbS3ScaU6Gn00GaIpuFnlFsyx NwXT/J/G0Hyn9XXFrThS9oH5/AkSGeLFYjKJIOUEcAgpg1/Q7QXdlrvzaMz6XXu6qzBW 1BR726SOBvTsepVgnyKf93FCxR9AnBQ4lqBGvMYQxVh/R3TjTXsFhSiI2b8d/hlbwsYn U9qUW0aNjLlWbPD/jPuqE2DvNJLQRmd5R+mO4vJclGmTringVj56M0yBjYLMt0k8fmxP ooXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:cc:content-type; bh=AsB1xKHKpLw8KObYUYvj9O1np6ls5yLXy4gh3kLhAfw=; b=EIS6t0jhR8Hl3vnrbxhLfmmGQWbN+J18k+/7u6si97UOnDqPvXvJiAkd7H9DHRWWfR +jNocqX2RkF8o04+bmexZODc2q9INZEDBmOlvg5fOTrL9X5o01wke13ymfY8I+X3F9Nu Fc7ay3Rb4ufH29E7s2BI9RKJM008CU0qPx4HVjLpVD/Cg34BVvW2EjlhYRPjpEOM9loW OnoDghNn4uG3zu1zX3xo4PQRRHJzPli9MmPu6ALHtF67aeP+nj9GJSTpw1HKzpQkvpBp U08+h3kFjYp16Gsx79EkeM3w6ZGS62v6zhqnQOpykkBM0U/L+N3gMLBwK8y9mDsbJuw9 dhRA== X-Gm-Message-State: AG10YOS0vtmDUKCYQP/fw4sB1HxLF+mdxnFMlDevCHFuOSC0bxpmp/6H/QLprtwykkD+OqXU+rXR4QoF8YC6aQ== X-Received: by 10.50.111.169 with SMTP id ij9mr2662112igb.92.1453978610975; Thu, 28 Jan 2016 02:56:50 -0800 (PST) In-Reply-To: X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.20+ Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Xref: csiph.com comp.lang.python:102177 On Thu, Jan 28, 2016 at 8:13 PM, Frank Millman wrote: > Run the database handler in a separate thread. Use a queue.Queue to send > requests to the handler. Use an asyncio.Queue to send results back to the > caller, which can call 'await q.get()'. > > I ran a quick test and it seems to work. What do you think? My gut feeling is that any queue can block at either get or put. Half of your operations are "correct", and the other half are "wrong". The caller can "await q.get()", which is the "correct" way to handle the blocking operation; the database thread can "jobqueue.get()" as a blocking operation, which is also fine. But your queue-putting operations are going to have to assume that they never block. Maybe you can have the database put its results onto the asyncio.Queue safely, but the requests going onto the queue.Queue could block waiting for the database. Specifically, this will happen if database queries come in faster than the database can handle them - quite literally, your jobs will be "blocked on the database". What should happen then? The other risk is that the wrong result will be queried (two async tasks put something onto the queue - which one gets the first result?), which could either be coped with by simple sequencing (maybe this happens automatically, although I'd prefer a mathematically-provable result to "it seems to work"), or by wrapping the whole thing up in a function/class. Both of these are risks seen purely by looking at the idea description, not at any sort of code. It's entirely possible I'm mistaken about them. But there's only one way to find out :) By the way, just out of interest... there's no way you can actually switch out the database communication for something purely socket-based, is there? PostgreSQL's protocol, for instance, is fairly straight-forward, and you don't *have* to use libpq; Pike's inbuilt pgsql module just opens a socket and says hello, and it looks like py-postgresql [1] is the same thing for Python. Taking something like that and making it asynchronous would be as straight-forward as converting any other socket-based code. Could be an alternative to all this weirdness. ChrisA [1] https://pypi.python.org/pypi/py-postgresql