Re: python - handling HTTP requests asynchronously

Path	csiph.com!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From	Michael Selik <michael.selik@gmail.com>
Newsgroups	comp.lang.python
Subject	Re: python - handling HTTP requests asynchronously
Date	Sat, 07 May 2016 19:04:21 +0000
Lines	27
Message-ID	<mailman.462.1462647873.32212.python-list@python.org> (permalink)
References	<dc385b5f-a453-441c-9476-ea73515201ad@googlegroups.com> <CAGgTfkO4Gm-JkjY6Fu+O0yO6Se4RSAzA6XUxp4mF_qLnXGKAvA@mail.gmail.com>
Mime-Version	1.0
Content-Type	text/plain; charset=UTF-8
X-Trace	news.uni-berlin.de CHwoZ+RCZzc/+5+2ubNCAg024fmzcG7TBDWDralqOUEA==
Return-Path	<michael.selik@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.001
X-Spam-Evidence	'H': 1.00; 'S': 0.00; 'api.': 0.04; 'concurrently': 0.07; 'wrapper': 0.07; 'api': 0.09; 'concurrent': 0.09; 'retry': 0.09; 'subject:skip:a 10': 0.09; 'threads.': 0.09; 'subject:python': 0.14; '2016': 0.16; 'backoff': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; 'subject:handling': 0.16; 'subject:requests': 0.16; 'threads': 0.16; 'wrote:': 0.16; 'module,': 0.18; 'email addr:gmail.com>': 0.18; 'to:21': 0.21; 'header:In-Reply-To:1': 0.24; 'requests': 0.25; 'external': 0.27; 'skip:m 30': 0.27; '(e.g.': 0.27; 'fri,': 0.27; 'to:no real name:21': 0.27; 'message-id:@mail.gmail.com': 0.27; 'interface': 0.29; 'handled': 0.29; 'code': 0.30; 'e.g.': 0.30; 'http': 0.33; 'received:google.com': 0.35; 'but': 0.36; 'too': 0.36; 'received:209.85': 0.36; 'to:addr:python-list': 0.36; 'subject:: ': 0.37; 'pdf': 0.37; 'doing': 0.38; 'received:209': 0.38; 'skip:s 40': 0.38; 'received:209.85.220': 0.38; 'does': 0.39; 'to:addr:python.org': 0.40; 'easy': 0.60; 'your': 0.60; 'goal': 0.64; 'sounds': 0.76; '100': 0.79; 'sending,': 0.84
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=6INDHF31nl2qilvckDVrNfYjr2bNk0p0BioEQVQaLkY=; b=uUDcibimuwX7aV36qe+4T3x5eE8cb2nMYab+L72SwInGiHaERbPOICil3MxiQLFSAv yYJ8wxEXPfUOFmbYpC4MZg2Hy4+xCGe17z/WdznNmnNsqsz66N37YuosSPuumK0UA1uQ K7DGEldSItKb7rsQGEPiV+QANo4uG0eOmVrVdvzgmqSHRIMkBxcf7KQFc4p03UEtSA3B /59aWlrNeWFvrK3jMwzd5/nx135QXdLuvabD2934lAAme1LfKbzd2yqABB6f0ygzc41D IRzIInqVuXqfHydvhsxtkUaqzn6AXR5whrqOCgHZSm9LPXuz+m4OSvMkrHRjPyobFzPS NKPg==
X-Google-DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=6INDHF31nl2qilvckDVrNfYjr2bNk0p0BioEQVQaLkY=; b=R7kO/tSGEpk1mrAijkC1QbC4X82pzMnO8EGkX3Bao8KEfLjEIx4YIdtIGNRMSRTdJS kr2XGzeyT4uVzy+z4M3B7PeAkUr8berkujFxela2uKb1GMKcAKuJOBCHx2DAGlBosRfW esmCkQE4H5OPIOsnULjCZftnUS+xsKguV5g23AASQIlqDXjOAobO0sExC1onW93NHWE1 HnLtIX6DcLPCgw5ld95p3DdV/pAwT3ai+42uWzKMsoRzadc+jmQYk0/Szu9oPcFuLzhL 8bQeSpOILKmuU4lomC6ooeyNmCy7txF/DTIj6oD7Jo7g+MPIYIq9wB/rN/k/V9tHq8fV y0hw==
X-Gm-Message-State	AOPr4FX9LC7DHG70D6jH/c111QtnKlvp2VHITrEMKkPQ+1wUkVaeyaTFjcvJfq2dOBW0HqfY+0nDrQ3Mab6QyQ==
X-Received	by 10.55.68.78 with SMTP id r75mr26077743qka.189.1462647870389; Sat, 07 May 2016 12:04:30 -0700 (PDT)
In-Reply-To	<dc385b5f-a453-441c-9476-ea73515201ad@googlegroups.com>
X-Content-Filtered-By	Mailman/MimeDel 2.1.22
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.22
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID	<CAGgTfkO4Gm-JkjY6Fu+O0yO6Se4RSAzA6XUxp4mF_qLnXGKAvA@mail.gmail.com>
X-Mailman-Original-References	<dc385b5f-a453-441c-9476-ea73515201ad@googlegroups.com>
Xref	csiph.com comp.lang.python:108283

Show key headers only | View raw

On Fri, May 6, 2016 at 3:01 AM <lordluke80@gmail.com> wrote:

> The PDF is generated through an external API. Since currently is generated
> on demand, this is handled synchronously via an HTTP request/response.

Are you sending the request or are you receiving the request?
If you are sending, you can just use threads as you are only doing IO.
If you are receiving the requests and generating PDFs, you may want to use
subprocesses if the PDF-generation is compute-intensive.

> 3) multiprocessing module, with e.g. 10 as workers limit.
>

multiprocessing.Pool is an easy way to use subprocesses
multiprocessing.pool.ThreadPool is an easy way to use threads. It's not
well documented, but has the exact same interface as Pool.

the goal is to use the API concurrently (e.g. send 10 or 100 http requests
> simultaneously, depending on how many concurrent requests the API can
> handle).
>

Sounds like you want to use threads. How does the API let you know you're
hitting it too frequently? Perhaps you want to code an exponential backoff
and retry wrapper for your API requests.

Thread

python - handling HTTP requests asynchronously lordluke80@gmail.com - 2016-05-05 23:56 -0700
  Re: python - handling HTTP requests asynchronously justin walters <walters.justin01@gmail.com> - 2016-05-06 08:23 -0700
  Re: python - handling HTTP requests asynchronously Michael Selik <michael.selik@gmail.com> - 2016-05-07 19:04 +0000
    Re: python - handling HTTP requests asynchronously lordluke80@gmail.com - 2016-05-07 14:42 -0700
      Re: python - handling HTTP requests asynchronously Michael Selik <michael.selik@gmail.com> - 2016-05-10 23:54 +0000
      Re: python - handling HTTP requests asynchronously edgargdcv@gmail.com - 2016-07-19 11:59 -0700

csiph-web