Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #101246 > unrolled thread

What is the fastest way to do 400 HTTP requests using requests library?

Back to article view | Back to comp.lang.python

Contents

  What is the fastest way to do 400 HTTP requests using requests library? livemsn22@gmail.com - 2016-01-04 12:50 -0800
    Re: What is the fastest way to do 400 HTTP requests using requests library? Steven D'Aprano <steve@pearwood.info> - 2016-01-05 10:38 +1100
      Re: What is the fastest way to do 400 HTTP requests using requests library? Ian Kelly <ian.g.kelly@gmail.com> - 2016-01-04 16:51 -0700
      Re: What is the fastest way to do 400 HTTP requests using requests library? Tony van der Hoff <tony@vanderhoff.org> - 2016-01-05 09:53 +0100
        Re: What is the fastest way to do 400 HTTP requests using requests library? Steven D'Aprano <steve@pearwood.info> - 2016-01-05 20:38 +1100
          Re: What is the fastest way to do 400 HTTP requests using requests library? Tim Chase <python.list@tim.thechases.com> - 2016-01-05 05:43 -0600
          Re: What is the fastest way to do 400 HTTP requests using requests library? Paul Rubin <no.email@nospam.invalid> - 2016-01-05 09:02 -0800
            Re: What is the fastest way to do 400 HTTP requests using requests library? Ian Kelly <ian.g.kelly@gmail.com> - 2016-01-05 10:32 -0700

#101246 — What is the fastest way to do 400 HTTP requests using requests library?

So what is the fastest way to make 400 HTTP requests using "requests" library and also using tor proxy?

Best regards

[toc] | [next] | [standalone]

#101249

On Tue, 5 Jan 2016 07:50 am, livemsn22@gmail.com wrote:

> So what is the fastest way to make 400 HTTP requests using "requests"
> library and also using tor proxy?


Since this will be I/O bound, not CPU bound, probably use separate threads.

Push the 400 requests into a queue, then create N threads, where N will need
to be determined by experiment, but will probably be something like 4 or 8,
and let each thread pop a request from the queue as needed.

Are you experienced with threads? Do you need further information about
using threads and queues?




-- 
Steven

[toc] | [prev] | [next] | [standalone]

#101250

On Mon, Jan 4, 2016 at 4:38 PM, Steven D'Aprano <steve@pearwood.info> wrote:
> On Tue, 5 Jan 2016 07:50 am, livemsn22@gmail.com wrote:
>
>> So what is the fastest way to make 400 HTTP requests using "requests"
>> library and also using tor proxy?
>
>
> Since this will be I/O bound, not CPU bound, probably use separate threads.
>
> Push the 400 requests into a queue, then create N threads, where N will need
> to be determined by experiment, but will probably be something like 4 or 8,
> and let each thread pop a request from the queue as needed.
>
> Are you experienced with threads? Do you need further information about
> using threads and queues?

Also see the concurrent.futures module in the standard library, which
makes this sort of setup very simple to implement.

[toc] | [prev] | [next] | [standalone]

#101263

On 05/01/16 00:51, Ian Kelly wrote:
> On Mon, Jan 4, 2016 at 4:38 PM, Steven D'Aprano <steve@pearwood.info> wrote:
>> On Tue, 5 Jan 2016 07:50 am, livemsn22@gmail.com wrote:
>>
>>> So what is the fastest way to make 400 HTTP requests using "requests"
>>> library and also using tor proxy?
>>
>>
>> Since this will be I/O bound, not CPU bound, probably use separate threads.
>>
>> Push the 400 requests into a queue, then create N threads, where N will need
>> to be determined by experiment, but will probably be something like 4 or 8,
>> and let each thread pop a request from the queue as needed.
>>
>> Are you experienced with threads? Do you need further information about
>> using threads and queues?
> 
> Also see the concurrent.futures module in the standard library, which
> makes this sort of setup very simple to implement.
> 
Why would someone want to make 400 HTTP requests in a short time?


-- 
Tony van der Hoff  | mailto:tony@vanderhoff.org
Ariège, France     |

[toc] | [prev] | [next] | [standalone]

#101264

On Tue, 5 Jan 2016 07:53 pm, Tony van der Hoff wrote:

> Why would someone want to make 400 HTTP requests in a short time?

For the same reason they want to make 400 HTTP requests over a long time,
except that they're in a hurry.

Maybe they're stress-testing a web server, or they just want to download
things in a rush.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#101269

On 2016-01-05 20:38, Steven D'Aprano wrote:
> On Tue, 5 Jan 2016 07:53 pm, Tony van der Hoff wrote:
> 
> > Why would someone want to make 400 HTTP requests in a short time?
> 
> For the same reason they want to make 400 HTTP requests over a long
> time, except that they're in a hurry.
> 
> Maybe they're stress-testing a web server, or they just want to
> download things in a rush.

Maybe they just want to generate lots of errors.  It's generally more
productive to make lots of 200 HTTP requests[*]. ;-)

-tkc


[*] http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

[toc] | [prev] | [next] | [standalone]

#101273

Steven D'Aprano <steve@pearwood.info> writes:
> Maybe they're stress-testing a web server, or they just want to download
> things in a rush.

They're stress-testing a web server through a tor proxy?  This sounds
abusive to me.

I also wonder whether 400 referred to the HTTP 400 error code rather
than the number of requests to be sent.  As in:

    - Layer 7 (“400 bad request”) attacks toward our web and application
      servers, causing Linode Manager outages

from http://status.linode.com/incidents/mmdbljlglnfd
regarding a big DDOS attack that's been running against Linode.com
(a VPS host) over the past couple weeks.

[toc] | [prev] | [next] | [standalone]

#101274

On Tue, Jan 5, 2016 at 10:02 AM, Paul Rubin <no.email@nospam.invalid> wrote:
> Steven D'Aprano <steve@pearwood.info> writes:
>> Maybe they're stress-testing a web server, or they just want to download
>> things in a rush.
>
> They're stress-testing a web server through a tor proxy?  This sounds
> abusive to me.
>
> I also wonder whether 400 referred to the HTTP 400 error code rather
> than the number of requests to be sent.  As in:
>
>     - Layer 7 (“400 bad request”) attacks toward our web and application
>       servers, causing Linode Manager outages
>
> from http://status.linode.com/incidents/mmdbljlglnfd
> regarding a big DDOS attack that's been running against Linode.com
> (a VPS host) over the past couple weeks.

I had the same initial thought about the status code but dismissed it,
since why would somebody intentionally send a request that will return
a 400 status? I was not thinking about abuse though, so the DDoS
scenario did not occur to me.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web