Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #45525

Re: Please help with Threading

Show all headers | View raw

On Sat, 18 May 2013 01:58:13 -0700 (PDT), Jurgens de Bruin
<debruinjj@gmail.com> declaimed the following in
gmane.comp.python.general:

> This is my first script where I want to use the python threading module. I have a large dataset which is a list of dict this can be as much as 200 dictionaries in the list. The final goal is a  histogram for each dict 16 histograms on a page ( 4x4 ) - this already works. 
> What I currently do is a create a nested list [ [ {}  ], [ {} ] ] each inner list contains 16 dictionaries, thus each inner list is a single page of 16 histograms. Iterating over the outer-list  and creating the graphs takes to long. So I would like multiple inner-list to be processes simultaneously and creating the graphs in "parallel". 
> I am trying to use the python threading for this. I create 4 threads loop over the outer-list and send a inner-list to the thread. This seems to work if my nested lists only contains 2 elements - thus less elements than threads. Currently the scripts runs and then seems to get hung up. I monitor the resource  on my mac and python starts off good using 80% and when the 4-thread is created the CPU usages drops to 0%. 
> 

	The odds are good that this is just going to run slower...

	One: The common Python implementation uses a global interpreter lock
to prevent interpreted code from interfering with itself in multiple
threads. So "number cruncher" applications don't gain any speed from
being partitioned into thread -- even on a multicore processor, only one
thread can have the GIL at a time. On top of that, you have the overhead
of the interpreter switching between threads (GIL release on one thread,
GIL acquire for the next thread).

	Python threads work fine if the threads either rely on intelligent
DLLs for number crunching (instead of doing nested Python loops to
process a numeric array you pass it to something like NumPy which
releases the GIL while crunching a copy of the array) or they do lots of
I/O and have to wait for I/O devices (while one thread is waiting for
the write/read operation to complete, another thread can do some number
crunching).

	If you really need to do this type of number crunching in Python
level code, you'll want to look into the multiprocessing library
instead. That will create actual OS processes (each with a copy of the
interpreter, and not sharing memory) and each of those can run on a core
without conflicting on the GIL.
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Please help with Threading Jurgens de Bruin <debruinjj@gmail.com> - 2013-05-18 01:58 -0700
  Re: Please help with Threading Peter Otten <__peter__@web.de> - 2013-05-18 11:09 +0200
    Re: Please help with Threading Jurgens de Bruin <debruinjj@gmail.com> - 2013-05-18 04:23 -0700
      Re: Please help with Threading Peter Otten <__peter__@web.de> - 2013-05-18 14:01 +0200
  Re: Please help with Threading Dave Angel <davea@davea.name> - 2013-05-18 09:55 -0400
  Re: Please help with Threading Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-05-18 15:28 -0400
  RE: Please help with Threading Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-19 03:02 +0300
  Re: Please help with Threading Chris Angelico <rosuav@gmail.com> - 2013-05-19 10:38 +1000
  Re: Please help with Threading Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-05-19 17:46 -0400
  Re: Please help with Threading Chris Angelico <rosuav@gmail.com> - 2013-05-20 07:52 +1000
  Re: Please help with Threading Dave Angel <davea@davea.name> - 2013-05-19 21:04 -0400
  Re: Please help with Threading Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-05-19 22:58 -0400
  Re: Please help with Threading Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-05-19 23:25 -0400
  Re: Please help with Threading Fábio Santos <fabiosantosart@gmail.com> - 2013-05-20 07:25 +0100
  Re: Please help with Threading Jurgens de Bruin <debruinjj@gmail.com> - 2013-06-02 18:47 -0700

csiph-web