Path: csiph.com!usenet.pasdenom.info!dedibox.gegeweb.org!gegeweb.eu!nntpfeed.proxad.net!proxad.net!feeder1-2.proxad.net!usenet-fr.net!de-l.enfer-du-nord.net!feeder1.enfer-du-nord.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Newsgroups: comp.lang.python
Date: Mon, 3 Sep 2012 15:09:26 -0700 (PDT)
In-Reply-To: <mailman.37.1346527731.27098.python-list@python.org>
Complaints-To: groups-abuse@google.com
Injection-Info: glegroupsg2000goo.googlegroups.com; posting-host=174.252.48.95; posting-account=vglGDQoAAABmaDKgNY6lPt_hwJvC5hhE
References: <50419A55.3080904@shopzeus.com> <5042434B.3010201@thieprojects.ch> <mailman.37.1346527731.27098.python-list@python.org>
User-Agent: G2/1.0
MIME-Version: 1.0
Subject: Re: Async client for PostgreSQL?
From: calderone.jeanpaul@gmail.com
To: comp.lang.python@googlegroups.com
Content-Type: text/plain; charset=ISO-8859-1
Cc: python-list@python.org
Precedence: list
Message-ID: <mailman.159.1346710169.27098.python-list@python.org>
Lines: 117
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:28374

On Saturday, September 1, 2012 3:28:52 PM UTC-4, Laszlo Nagy wrote:
> > Hi
> 
> >
> 
> > does running on tornado imply that you would not consider twisted 
> 
> > http://twistedmatrix.com ?
> 
> >
> 
> > If not, twisted has exactly this capability hiding long running 
> 
> > queries on whatever db's behind deferToThread().
> 
> All right, I was reading its documentation
> 
> 
> 
> http://twistedmatrix.com/documents/10.1.0/api/twisted.internet.threads.deferToThread.html
> 
> 
> 
> It doesn't tell too much about it: "Run a function in a thread and 
> 
> return the result as a Deferred.".
> 

You can find more documentation here:

http://twistedmatrix.com/documents/current/core/howto/threading.html

Also, Twisted has dedicated APIs for interacting with databases asynchronously:

http://twistedmatrix.com/documents/current/core/howto/rdbms.html

Additionally, there is a non-blocking (rather than thread-based) implementation of the above API available for PostgreSQL:

http://pypi.python.org/pypi/txpostgres

> 
> 
> Run a function but in what thread? Does it create a new thread for every 
> 
> invocation? In that case, I don't want to use this. My example case: 10% 
> 
> from 100 requests/second deal with a database. But it does not mean that 
> 
> one db-related request will do a single db API call only. They will 
> 
> almost always do more: start transaction, parse and open query, fetch 
> 
> with cursor, close query, open another query etc. then commit 
> 
> transaction. 8 API calls to do a quick fetch + update (usually under 
> 
> 100msec, but it might be blocked by another transaction for a while...) 
> 
> So we are talking about 80 database API calls per seconds at least. It 
> 
> would be insane to initialize a new thread for each invocation. And 
> 
> wrapping these API calls into a single closure function is not useful 
> 
> either, because that function would not be able to safely access the 
> 
> state that is stored in the main thread. Unless you protet it with 
> 
> locks. But it is whole point of async I/O server: to avoid using slow 
> 
> locks, expensive threads and context switching.
> 
> 
> 
> Maybe, deferToThread uses a thread pool? But it doesn't say much about 
> 
> it. (Am I reading the wrong documentation?) BTW I could try a version 
> 
> that uses a thread pool.
> 
> 
> 
> It is sad, by the way. We have async I/O servers for Python that can be 
> 
> used for large number of clients, but most external modules/extensions 
> 
> do not support their I/O loops. Including the extension modules of the 
> 
> most popular databases. So yes, you can use Twisted or torandoweb until 
> 
> you do not want to call *some* API functions that are blocking. (By 
> 
> *some* I mean: much less blocking than non-blocking, but quite a few.) 
> 
> We also have synchronous Python servers, but we cannot get rid of the 
> 
> GIL, Python threads are expensive and slow, so they cannot be used for a 
> 
> large number of clients. And finally, we have messaging services/IPC 
> 
> like zeromq. They are probably the most expensive, but they scale very 
> 
> well. But you need more money to operate the underlying hardware. I'm 
> 
> starting to think that I did not get a quick answer because my use case 
> 
> (100 clients) fall into to the "heavy weight" category, and the solution 
> 
> is to invest more in the hardware. :-)
> 
> 
> 
> Thanks,
> 
> 
> 
>     Laszlo