Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #101263

Re: What is the fastest way to do 400 HTTP requests using requests library?

Show all headers | View raw

On 05/01/16 00:51, Ian Kelly wrote:
> On Mon, Jan 4, 2016 at 4:38 PM, Steven D'Aprano <steve@pearwood.info> wrote:
>> On Tue, 5 Jan 2016 07:50 am, livemsn22@gmail.com wrote:
>>
>>> So what is the fastest way to make 400 HTTP requests using "requests"
>>> library and also using tor proxy?
>>
>>
>> Since this will be I/O bound, not CPU bound, probably use separate threads.
>>
>> Push the 400 requests into a queue, then create N threads, where N will need
>> to be determined by experiment, but will probably be something like 4 or 8,
>> and let each thread pop a request from the queue as needed.
>>
>> Are you experienced with threads? Do you need further information about
>> using threads and queues?
> 
> Also see the concurrent.futures module in the standard library, which
> makes this sort of setup very simple to implement.
> 
Why would someone want to make 400 HTTP requests in a short time?


-- 
Tony van der Hoff  | mailto:tony@vanderhoff.org
Ariège, France     |

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

What is the fastest way to do 400 HTTP requests using requests library? livemsn22@gmail.com - 2016-01-04 12:50 -0800
  Re: What is the fastest way to do 400 HTTP requests using requests library? Steven D'Aprano <steve@pearwood.info> - 2016-01-05 10:38 +1100
    Re: What is the fastest way to do 400 HTTP requests using requests library? Ian Kelly <ian.g.kelly@gmail.com> - 2016-01-04 16:51 -0700
    Re: What is the fastest way to do 400 HTTP requests using requests library? Tony van der Hoff <tony@vanderhoff.org> - 2016-01-05 09:53 +0100
      Re: What is the fastest way to do 400 HTTP requests using requests library? Steven D'Aprano <steve@pearwood.info> - 2016-01-05 20:38 +1100
        Re: What is the fastest way to do 400 HTTP requests using requests library? Tim Chase <python.list@tim.thechases.com> - 2016-01-05 05:43 -0600
        Re: What is the fastest way to do 400 HTTP requests using requests library? Paul Rubin <no.email@nospam.invalid> - 2016-01-05 09:02 -0800
          Re: What is the fastest way to do 400 HTTP requests using requests library? Ian Kelly <ian.g.kelly@gmail.com> - 2016-01-05 10:32 -0700

csiph-web