Re: What is the fastest way to do 400 HTTP requests using requests library?

Path	csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From	Ian Kelly <ian.g.kelly@gmail.com>
Newsgroups	comp.lang.python
Subject	Re: What is the fastest way to do 400 HTTP requests using requests library?
Date	Mon, 4 Jan 2016 16:51:16 -0700
Lines	18
Message-ID	<mailman.7.1451951524.2305.python-list@python.org> (permalink)
References	<0e42a90b-b736-4050-a20c-6d387048daf3@googlegroups.com> <568b027c$0$1588$c3e8da3$5496439d@news.astraweb.com>
Mime-Version	1.0
Content-Type	text/plain; charset=UTF-8
X-Trace	news.uni-berlin.de vxS+S7rphPJITZT+h/WBfQwIFmCqXhk6Tgz/vXo3qw8A==
Return-Path	<ian.g.kelly@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.002
X-Spam-Evidence	'H': 1.00; 'S': 0.00; 'pop': 0.05; 'subject:library': 0.09; 'subject:using': 0.09; 'threads,': 0.09; 'threads.': 0.09; 'thread': 0.10; 'jan': 0.11; '2016': 0.16; 'bound,': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; 'subject:fastest': 0.16; 'subject:requests': 0.16; 'threads': 0.16; 'wrote:': 0.16; 'library,': 0.18; 'library': 0.20; 'am,': 0.23; 'needed.': 0.23; 'header:In-Reply-To:1': 0.24; 'mon,': 0.24; 'requests': 0.25; 'sort': 0.25; 'module': 0.25; 'separate': 0.27; 'message-id:@mail.gmail.com': 0.27; 'fastest': 0.27; 'cpu': 0.29; 'i/o': 0.29; 'queue': 0.29; 'push': 0.30; 'probably': 0.31; "d'aprano": 0.33; 'http': 0.33; 'steven': 0.33; 'tue,': 0.34; 'received:google.com': 0.35; 'something': 0.35; 'but': 0.36; 'received:209.85': 0.36; 'to:addr:python-list': 0.36; 'subject:?': 0.36; 'pm,': 0.36; 'subject:: ': 0.37; 'received:209.85.213': 0.37; 'received:209': 0.38; 'subject:the': 0.39; 'to:addr:python.org': 0.40; 'where': 0.40; 'email addr:gmail.com': 0.62; 'further': 0.62; 'information': 0.63; 'to:name:python': 0.84
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=afizSb5uRyhArYGcRKVhPW702fVNrIrAHp2cLS1GST4=; b=Zy8+IbGYkU0yaKJxQKAO0L3uiYQ5pUKQJp8V6DtXraY8rwWdwl0+bd2J3L+YXvTwKR Qw6+0MKewwQYI4zWk8xw6J5bvzg+HnbLYxyHzQ4PKgfVSsW4gHFjEd+lc0XVSxZJ/yky uFlfZys1Iq4vgMmmF5DPpWccL3cEOrY433N0v4O9OQhT4fbE68nEYRBcl8LqOq2zefAu CcjhdRA6GATmAxNVah+/cOPR9zD8f0Q5w0m1mKhQTZ7c9vnTx9mB9w/5EYm9D3qRnchU P+b+AoGNSmR99WY+OFJNx9gL+svovUYcgwDjDI4tdWp6reMd8gDI/3RlqIyIYnnjSiQG zHyw==
X-Received	by 10.50.65.39 with SMTP id u7mr971144igs.85.1451951515908; Mon, 04 Jan 2016 15:51:55 -0800 (PST)
In-Reply-To	<568b027c$0$1588$c3e8da3$5496439d@news.astraweb.com>
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.20+
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Xref	csiph.com comp.lang.python:101250

Show key headers only | View raw

On Mon, Jan 4, 2016 at 4:38 PM, Steven D'Aprano <steve@pearwood.info> wrote:
> On Tue, 5 Jan 2016 07:50 am, livemsn22@gmail.com wrote:
>
>> So what is the fastest way to make 400 HTTP requests using "requests"
>> library and also using tor proxy?
>
>
> Since this will be I/O bound, not CPU bound, probably use separate threads.
>
> Push the 400 requests into a queue, then create N threads, where N will need
> to be determined by experiment, but will probably be something like 4 or 8,
> and let each thread pop a request from the queue as needed.
>
> Are you experienced with threads? Do you need further information about
> using threads and queues?

Also see the concurrent.futures module in the standard library, which
makes this sort of setup very simple to implement.

Thread

What is the fastest way to do 400 HTTP requests using requests library? livemsn22@gmail.com - 2016-01-04 12:50 -0800
  Re: What is the fastest way to do 400 HTTP requests using requests library? Steven D'Aprano <steve@pearwood.info> - 2016-01-05 10:38 +1100
    Re: What is the fastest way to do 400 HTTP requests using requests library? Ian Kelly <ian.g.kelly@gmail.com> - 2016-01-04 16:51 -0700
    Re: What is the fastest way to do 400 HTTP requests using requests library? Tony van der Hoff <tony@vanderhoff.org> - 2016-01-05 09:53 +0100
      Re: What is the fastest way to do 400 HTTP requests using requests library? Steven D'Aprano <steve@pearwood.info> - 2016-01-05 20:38 +1100
        Re: What is the fastest way to do 400 HTTP requests using requests library? Tim Chase <python.list@tim.thechases.com> - 2016-01-05 05:43 -0600
        Re: What is the fastest way to do 400 HTTP requests using requests library? Paul Rubin <no.email@nospam.invalid> - 2016-01-05 09:02 -0800
          Re: What is the fastest way to do 400 HTTP requests using requests library? Ian Kelly <ian.g.kelly@gmail.com> - 2016-01-05 10:32 -0700

csiph-web