Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #72505

Re: Benefits of asyncio

Show all headers | View raw

"Chris Angelico" <rosuav@gmail.com> wrote in message 
news:CAPTjJmqWkEStvrsrg30qjO+4TtLqfK9Q4GaByGovEw8NsdXzPg@mail.gmail.com...
>
> This works as long as your database is reasonably fast and close
> (common case for a lot of web servers: DB runs on same computer as web
> and application and etc servers). It's nice and simple, lets you use a
> single database connection (although you should probably wrap it in a
> try/finally to ensure that you roll back on any exception), and won't
> materially damage throughput as long as you don't run into problems.
> For a database driven web site, most of the I/O time will be waiting
> for clients, not waiting for your database.
>
> Getting rid of those blocking database calls means having multiple
> concurrent transactions on the database. Whether you go async or
> threaded, this is going to happen. Unless your database lets you run
> multiple simultaneous transactions on a single connection (I don't
> think the Python DB API allows that, and I can't think of any DB
> backends that support it, off hand), that means that every single
> concurrency point needs its own database connection. With threads, you
> could have a pool of (say) a dozen or so, one per thread, with each
> one working synchronously; with asyncio, you'd have to have one for
> every single incoming client request, or else faff around with
> semaphores and resource pools and such manually. The throughput you
> gain by making those asynchronous with callbacks is quite probably
> destroyed by the throughput you lose in having too many simultaneous
> connections to the database. I can't prove that, obviously, but I do
> know that PostgreSQL requires up-front RAM allocation based on the
> max_connections setting, and trying to support 5000 connections
> started to get kinda stupid.
>

I am following this with interest.  I still struggle to get my head around 
the concepts, but it is slowly coming clearer.

Focusing on PostgreSQL, couldn't you do the following?

PostgreSQL runs client/server (they call it front-end/back-end) over TCP/IP.

psycopg2 appears to have some support for async communication with the 
back-end. I only skimmed the docs, and it looks a bit complicated, but it is 
there.

So why not keep a 'connection pool', and for every potentially blocking 
request, grab a connection, set up a callback or a 'yield from' to wait for 
the response, and unblock.

Provided the requests return quickly, I would have thought a hundred 
database connections could support thousands of users.

Frank Millman

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Benefits of asyncio Aseem Bansal <asmbansal2@gmail.com> - 2014-06-02 10:40 -0700
  Re: Benefits of asyncio Ian Kelly <ian.g.kelly@gmail.com> - 2014-06-02 12:37 -0600
  Re: Benefits of asyncio Terry Reedy <tjreedy@udel.edu> - 2014-06-02 16:07 -0400
    Re: Benefits of asyncio Roy Smith <roy@panix.com> - 2014-06-02 16:19 -0400
    Re: Benefits of asyncio Marko Rauhamaa <marko@pacujo.net> - 2014-06-02 23:28 +0300
      Re: Benefits of asyncio Paul Rubin <no.email@nospam.invalid> - 2014-06-02 13:45 -0700
        Re: Benefits of asyncio Chris Angelico <rosuav@gmail.com> - 2014-06-03 07:49 +1000
        Re: Benefits of asyncio Terry Reedy <tjreedy@udel.edu> - 2014-06-02 21:51 -0400
        Re: Benefits of asyncio Marko Rauhamaa <marko@pacujo.net> - 2014-06-03 09:36 +0300
          Re: Benefits of asyncio Chris Angelico <rosuav@gmail.com> - 2014-06-03 18:47 +1000
            Re: Benefits of asyncio Marko Rauhamaa <marko@pacujo.net> - 2014-06-03 12:10 +0300
              Re: Benefits of asyncio Chris Angelico <rosuav@gmail.com> - 2014-06-03 19:30 +1000
                Re: Benefits of asyncio Marko Rauhamaa <marko@pacujo.net> - 2014-06-03 13:08 +0300
                Re: Benefits of asyncio Chris Angelico <rosuav@gmail.com> - 2014-06-03 20:23 +1000
                Re: Benefits of asyncio Marko Rauhamaa <marko@pacujo.net> - 2014-06-03 14:12 +0300
                Re: Benefits of asyncio Paul Rubin <no.email@nospam.invalid> - 2014-06-04 00:52 -0700
              Re: Benefits of asyncio Burak Arslan <burak.arslan@arskom.com.tr> - 2014-06-03 14:05 +0300
              Re: Benefits of asyncio Chris Angelico <rosuav@gmail.com> - 2014-06-03 21:57 +1000
              Re: Benefits of asyncio Burak Arslan <burak.arslan@arskom.com.tr> - 2014-06-04 08:10 +0300
              Re: Benefits of asyncio Chris Angelico <rosuav@gmail.com> - 2014-06-04 17:30 +1000
              Re: Benefits of asyncio Paul Rubin <no.email@nospam.invalid> - 2014-06-04 00:48 -0700
          Re: Benefits of asyncio "Frank Millman" <frank@chagford.com> - 2014-06-03 13:09 +0200
          Re: Benefits of asyncio Chris Angelico <rosuav@gmail.com> - 2014-06-03 22:01 +1000
            Re: Benefits of asyncio Marko Rauhamaa <marko@pacujo.net> - 2014-06-03 16:05 +0300
              Re: Benefits of asyncio Chris Angelico <rosuav@gmail.com> - 2014-06-03 23:31 +1000
                Re: Benefits of asyncio Marko Rauhamaa <marko@pacujo.net> - 2014-06-03 16:42 +0300
                Re: Benefits of asyncio Chris Angelico <rosuav@gmail.com> - 2014-06-03 23:49 +1000
                Re: Benefits of asyncio Marko Rauhamaa <marko@pacujo.net> - 2014-06-03 19:18 +0300
              Re: Benefits of asyncio Roy Smith <roy@panix.com> - 2014-06-03 11:40 -0400
        Re: Benefits of asyncio Paul Sokolovsky <pmiscml@gmail.com> - 2014-06-03 11:31 +0300
  Re: Benefits of asyncio Burak Arslan <burak.arslan@arskom.com.tr> - 2014-06-03 00:07 +0300
  Re: Benefits of asyncio Aseem Bansal <asmbansal2@gmail.com> - 2014-06-02 21:54 -0700

csiph-web