Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #85588 > unrolled thread

Async/Concurrent HTTP Requests

Back to article view | Back to comp.lang.python

Contents

  Async/Concurrent HTTP Requests Ari King <ari.brandeis.king@gmail.com> - 2015-02-12 08:37 -0800
    Re: Async/Concurrent HTTP Requests Zachary Ware <zachary.ware+pylist@gmail.com> - 2015-02-12 11:59 -0600
    Re: Async/Concurrent HTTP Requests Paul Rubin <no.email@nospam.invalid> - 2015-02-12 10:15 -0800
      Re: Async/Concurrent HTTP Requests Marko Rauhamaa <marko@pacujo.net> - 2015-02-12 20:55 +0200
        Re: Async/Concurrent HTTP Requests Paul Rubin <no.email@nospam.invalid> - 2015-02-12 10:57 -0800
          Re: Async/Concurrent HTTP Requests Marko Rauhamaa <marko@pacujo.net> - 2015-02-12 21:37 +0200

#85588 — Async/Concurrent HTTP Requests

Hi,

I'd like to query two (or more) RESTful APIs concurrently. What is the pythonic way of doing so? Is it better to use built in functions or are third-party packages? Thanks.

Best,
Ari

[toc] | [next] | [standalone]

#85600

On Thu, Feb 12, 2015 at 10:37 AM, Ari King <ari.brandeis.king@gmail.com> wrote:
> Hi,
>
> I'd like to query two (or more) RESTful APIs concurrently. What is the pythonic way of doing so? Is it better to use built in functions or are third-party packages? Thanks.

Have a look at asyncio (new in Python 3.4, available for 3.3 as the
'tulip' project) and possibly the aiohttp project, available on PyPI.
I'm using both for a current project, and they work very well.

-- 
Zach

[toc] | [prev] | [next] | [standalone]

#85601

Ari King <ari.brandeis.king@gmail.com> writes:
> I'd like to query two (or more) RESTful APIs concurrently. What is the
> pythonic way of doing so? Is it better to use built in functions or
> are third-party packages? Thanks.

The two basic approaches are event-based asynchronous i/o (there are
various packages for that) and threads.  There are holy wars over which
is better.  Event-driven i/o in Python 2.x was generally done with
callback-based packages like Twisted Matrix (www.twistedmatrix.com).  In
Python 3 there are some nicer mechanisms (coroutines) so the new asyncio
package may be easier to use than Twisted.  I haven't tried it yet.

[toc] | [prev] | [next] | [standalone]

#85605

Paul Rubin <no.email@nospam.invalid>:

> Event-driven i/o in Python 2.x was generally done with callback-based
> packages like Twisted Matrix (www.twistedmatrix.com). In Python 3
> there are some nicer mechanisms (coroutines) so the new asyncio
> package may be easier to use than Twisted. I haven't tried it yet.

I have successfully done event-driven I/O using select.epoll() and
socket.socket().


Marko

[toc] | [prev] | [next] | [standalone]

#85606

Marko Rauhamaa <marko@pacujo.net> writes:
> I have successfully done event-driven I/O using select.epoll() and
> socket.socket().

Sure, but then you end up writing a lot of low-level machinery that
packages like twisted take care of for you.

[toc] | [prev] | [next] | [standalone]

#85611

Paul Rubin <no.email@nospam.invalid>:

> Marko Rauhamaa <marko@pacujo.net> writes:
>> I have successfully done event-driven I/O using select.epoll() and
>> socket.socket().
>
> Sure, but then you end up writing a lot of low-level machinery that
> packages like twisted take care of for you.

Certainly. It would be nice if the stdlib protocol facilities were
event-driven and divorced from the low-level I/O.

Asyncio does that, of course, but the programming model feels a bit
weird.

Twisted documentation seems a bit vague on details. For example, what
should one make of this:

   def write(data):
      Write some data to the physical connection, in sequence, in a
      non-blocking fashion.

      If possible, make sure that it is all written. No data will ever
      be lost, although (obviously) the connection may be closed before
      it all gets through.

   <URL: https://twistedmatrix.com/documents/15.0.0/api/twisted.intern
   et.interfaces.ITransport.html#write>

So I'm left wondering if the call will block and if not, how is flow
control and buffering managed. The API documentation leads me to a maze
of twisted passages, all alike. From what I could gather, the write()
method is blocking and hence not suitable for serious work.

By contrast, the semantics of Python's socket.send() is crisply defined
and a pleasure to work with.


Marko

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web