Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #76675 > unrolled thread

Efficiency, threading, and concurrent.futures

Back to article view | Back to comp.lang.python

Contents

  Efficiency, threading, and concurrent.futures Rob Gaddi <rgaddi@technologyhighland.invalid> - 2014-08-20 09:43 -0700
    Re: Efficiency, threading, and concurrent.futures Akira Li <4kir4.1i@gmail.com> - 2014-08-20 21:17 +0400

#76675 — Efficiency, threading, and concurrent.futures

I've got a situation where I'll be asking an I/O bound process to do
some work (querying an RS-232 device) while my main code is off
running a sleep() bound process.  Everyone always talks about how
expensive thread creation is, so I figured I'd test it out in an
IPython notebook. 

#####

import threading
from concurent.futures import ThreadPoolExecutor as TPE
from time import sleep

def fn():
  sleep(0.001)

%%timeit -r 50 -n 1000
thr = threading.Thread(target=fn)
thr.start()
thr.join()
1000 loops, best of 5: 1.24 ms per loop

%%timeit -r 50 -n 1000 ex=TPE(1)
fut=ex.submit(fn)
fut.result()
1000 loops, best of 5: 1.26 ms per loop

#####

Now, my understanding is that the ThreadPoolExecutor spawns all its
threads at the outset, then stuffs requests into one queue and
fishes results out of another, which should be substantially faster than
having create new threads each time.  And yet those were pretty dead on
even. Any idea what I'm seeing here?

-- 
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order.  See above to fix.

[toc] | [next] | [standalone]

#76679

Rob Gaddi <rgaddi@technologyhighland.invalid> writes:

> I've got a situation where I'll be asking an I/O bound process to do
> some work (querying an RS-232 device) while my main code is off
> running a sleep() bound process.  Everyone always talks about how
> expensive thread creation is, so I figured I'd test it out in an
> IPython notebook. 
>
> #####
>
> import threading
> from concurent.futures import ThreadPoolExecutor as TPE
> from time import sleep
>
> def fn():
>   sleep(0.001)
>
> %%timeit -r 50 -n 1000
> thr = threading.Thread(target=fn)
> thr.start()
> thr.join()
> 1000 loops, best of 5: 1.24 ms per loop
>
> %%timeit -r 50 -n 1000 ex=TPE(1)
> fut=ex.submit(fn)
> fut.result()
> 1000 loops, best of 5: 1.26 ms per loop
>
> #####
>
> Now, my understanding is that the ThreadPoolExecutor spawns all its
> threads at the outset, then stuffs requests into one queue and
> fishes results out of another, which should be substantially faster than
> having create new threads each time.  And yet those were pretty dead on
> even. Any idea what I'm seeing here?

To see any difference, you should submit more than one job per worker to
ThreadPoolExecutor and avoid waiting for the each result synchronously.

I don't know whether ThreadPoolExecutor starts all workers at once in
the current CPython implementation. The name max_workers suggests that
it may start them as needed.


--
Akira

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web